

Dear xts experts
I use R and XTS to store dynamically growing price timeseries for currency
pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing with each new
incoming market price  say, one per second).
I want to speed up my R function that generates and rbinds a few new rows
to multiple Global XTS objects. This function us called often (sometimes
once per second) and takes a lot of time.
Specifically, I have a function add.newRate.EUR(POSIXct Time, float Rate)
This function performs some timeconsuming manipulations, then modifies
(global objects) XTS series1 through XTS seriesN. Specifically, I rbind
some new rows to each of series1 through seriesN in global environment.
(specific example  a new Rate for EUR/USD comes into the function. This
function needs to rbind new point to EUR.USD xts object, but it also needs
to calculate new value for crosscurrency pairs, like EUR/JPY, EUR/GPB,
EUR/CHF, EUR/CAD, EUR/AUD, EUR/NZD, and rbind new point to those XTS series
as well)
it all happens quite slowly in R. How do I accelerate it?
To avoid copying XTS series between the function and the global
environment, the function attempts to modify the seriesX in the global
environment, and not return XTS on function return.
The structure of my project is such that my other tasks in R need to access
these XTS series1 through seriesN in Global Environment.
Still, this whole setup is rather slow.
Do you think using Rcpp to move crosscurrency rate calculations to C++
could help?
E.g. I want to write add.newRate.EUR.RCPP(int Time, float Rate) in C++. I
think it will perform internal manipulations of time T and value X a little
faster.
Second idea for speed up: doing rbind in C++ using package RcppXts.
I plan to use the following function in RcppXts package:
function("xtsRbind",
&xtsRbind,
List::create(Named("x"), Named("y"), Named("dup")),
"Combine two xts objects rowwise");
Then I want to use Rcpp function assign( name, x ) to assign the created
object back to GlobalEnvironment.
Do you think the above way is a good way to grow dynamical XTS series inside
Rcpp?
Do you think using Rcpp and RcppXts will provide significant acceleration
compared to pure R ?
Thank you
Vlad
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


did some googling on my question
general wisdom on the internet seems to be  it's SLOW to grow R lists in a
loop. Must Preallocate First.
Question: is there a feature in XTS that allows to preallocate space for
growing XTS series?
thanks again!
On Fri, Sep 6, 2019 at 11:51 AM Vladimir Morozov < [hidden email]>
wrote:
> Dear xts experts
>
> I use R and XTS to store dynamically growing price timeseries for
> currency pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing with
> each new incoming market price  say, one per second).
>
> I want to speed up my R function that generates and rbinds a few new rows
> to multiple Global XTS objects. This function us called often (sometimes
> once per second) and takes a lot of time.
>
> Specifically, I have a function add.newRate.EUR(POSIXct Time, float Rate)
> This function performs some timeconsuming manipulations, then modifies
> (global objects) XTS series1 through XTS seriesN. Specifically, I rbind
> some new rows to each of series1 through seriesN in global environment.
> (specific example  a new Rate for EUR/USD comes into the function. This
> function needs to rbind new point to EUR.USD xts object, but it also needs
> to calculate new value for crosscurrency pairs, like EUR/JPY, EUR/GPB,
> EUR/CHF, EUR/CAD, EUR/AUD, EUR/NZD, and rbind new point to those XTS series
> as well)
>
> it all happens quite slowly in R. How do I accelerate it?
>
> To avoid copying XTS series between the function and the global
> environment, the function attempts to modify the seriesX in the global
> environment, and not return XTS on function return.
> The structure of my project is such that my other tasks in R need to
> access these XTS series1 through seriesN in Global Environment.
> Still, this whole setup is rather slow.
>
> Do you think using Rcpp to move crosscurrency rate calculations to C++
> could help?
> E.g. I want to write add.newRate.EUR.RCPP(int Time, float Rate) in C++. I
> think it will perform internal manipulations of time T and value X a little
> faster.
>
> Second idea for speed up: doing rbind in C++ using package RcppXts.
> I plan to use the following function in RcppXts package:
> function("xtsRbind",
> &xtsRbind,
> List::create(Named("x"), Named("y"), Named("dup")),
> "Combine two xts objects rowwise");
> Then I want to use Rcpp function assign( name, x ) to assign the created
> object back to GlobalEnvironment.
>
> Do you think the above way is a good way to grow dynamical XTS series inside
> Rcpp?
> Do you think using Rcpp and RcppXts will provide significant acceleration
> compared to pure R ?
>
> Thank you
> Vlad
>
>
>
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Hi Vladimir: I can tell that you tried to explain your problem clearly but
you'd get more
replies if you actually showed the R code that you wrote. Minimal,
reproducible examples
get a lot more responses. Good luck.
Mark
On Fri, Sep 6, 2019 at 12:05 AM Vladimir Morozov < [hidden email]>
wrote:
> did some googling on my question
> general wisdom on the internet seems to be  it's SLOW to grow R lists in a
> loop. Must Preallocate First.
>
> Question: is there a feature in XTS that allows to preallocate space for
> growing XTS series?
> thanks again!
>
> On Fri, Sep 6, 2019 at 11:51 AM Vladimir Morozov < [hidden email]>
> wrote:
>
> > Dear xts experts
> >
> > I use R and XTS to store dynamically growing price timeseries for
> > currency pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing
> with
> > each new incoming market price  say, one per second).
> >
> > I want to speed up my R function that generates and rbinds a few new rows
> > to multiple Global XTS objects. This function us called often (sometimes
> > once per second) and takes a lot of time.
> >
> > Specifically, I have a function add.newRate.EUR(POSIXct Time, float Rate)
> > This function performs some timeconsuming manipulations, then modifies
> > (global objects) XTS series1 through XTS seriesN. Specifically, I rbind
> > some new rows to each of series1 through seriesN in global environment.
> > (specific example  a new Rate for EUR/USD comes into the function. This
> > function needs to rbind new point to EUR.USD xts object, but it also
> needs
> > to calculate new value for crosscurrency pairs, like EUR/JPY, EUR/GPB,
> > EUR/CHF, EUR/CAD, EUR/AUD, EUR/NZD, and rbind new point to those XTS
> series
> > as well)
> >
> > it all happens quite slowly in R. How do I accelerate it?
> >
> > To avoid copying XTS series between the function and the global
> > environment, the function attempts to modify the seriesX in the global
> > environment, and not return XTS on function return.
> > The structure of my project is such that my other tasks in R need to
> > access these XTS series1 through seriesN in Global Environment.
> > Still, this whole setup is rather slow.
> >
> > Do you think using Rcpp to move crosscurrency rate calculations to C++
> > could help?
> > E.g. I want to write add.newRate.EUR.RCPP(int Time, float Rate) in C++. I
> > think it will perform internal manipulations of time T and value X a
> little
> > faster.
> >
> > Second idea for speed up: doing rbind in C++ using package RcppXts.
> > I plan to use the following function in RcppXts package:
> > function("xtsRbind",
> > &xtsRbind,
> > List::create(Named("x"), Named("y"), Named("dup")),
> > "Combine two xts objects rowwise");
> > Then I want to use Rcpp function assign( name, x ) to assign the created
> > object back to GlobalEnvironment.
> >
> > Do you think the above way is a good way to grow dynamical XTS series
> inside
> > Rcpp?
> > Do you think using Rcpp and RcppXts will provide significant acceleration
> > compared to pure R ?
> >
> > Thank you
> > Vlad
> >
> >
> >
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions
> should go.
>
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Wiadomość napisana przez Vladimir Morozov < [hidden email]> w dniu 06.09.2019, o godz. 04:51:
Dear xts experts
I use R and XTS to store dynamically growing price timeseries for currency pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing with each new incoming market price  say, one per second).
(…) it all happens quite slowly in R. How do I accelerate it?
1) forgotten xts acceleration:
to activate it add the DRBIND_APPEND flag to Makevars (linux/unix/macos) or Makevars.win (Windows), and recompile xts sources.
2) preallocation
preallocate_matrix < function(n) { x < matrix() length(x) < 4 * n # bid, ask, bid_size, ask_size dim(x) < c(n, 4) # see: ?dim return(x) }
> x < preallocate_matrix(5) > x [,1] [,2] [,3] [,4] [1,] NA NA NA NA [2,] NA NA NA NA [3,] NA NA NA NA [4,] NA NA NA NA [5,] NA NA NA NA
Now you need xts index, eg:
i < Sys.time() + 1:5 > i [1] "20190906 15:47:48 CEST" "20190906 15:47:49 CEST" "20190906 15:47:50 CEST" "20190906 15:47:51 CEST" [5] "20190906 15:47:52 CEST”
Our xts object:
> x < .xts(x, index = i) [,1] [,2] [,3] [,4] 20190906 15:47:48 NA NA NA NA 20190906 15:47:49 NA NA NA NA 20190906 15:47:50 NA NA NA NA 20190906 15:47:51 NA NA NA NA 20190906 15:47:52 NA NA NA NA
And now you can use this xts object without memory copying:
> x[1,] < c(1, 2, 3, 4) > x [,1] [,2] [,3] [,4] 20190906 15:47:48 1 2 3 4 20190906 15:47:49 NA NA NA NA 20190906 15:47:50 NA NA NA NA 20190906 15:47:51 NA NA NA NA 20190906 15:47:52 NA NA NA NA
> x[2,] < c(11, 22, 33, 44) > x [,1] [,2] [,3] [,4] 20190906 15:47:48 1 2 3 4 20190906 15:47:49 11 22 33 44 20190906 15:47:50 NA NA NA NA 20190906 15:47:51 NA NA NA NA 20190906 15:47:52 NA NA NA NA
etc.
That's what you meant.
Best, Daniel
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


> Wiadomość napisana przez Daniel Cegiełka < [hidden email]> w dniu 06.09.2019, o godz. 16:10:
>
>
> 2) preallocation
>
> preallocate_matrix < function(n)
> {
> x < matrix()
> length(x) < 4 * n # bid, ask, bid_size, ask_size
> dim(x) < c(n, 4) # see: ?dim
> return(x)
> }
>
> > x < preallocate_matrix(5)
> > x
> [,1] [,2] [,3] [,4]
> [1,] NA NA NA NA
> [2,] NA NA NA NA
> [3,] NA NA NA NA
> [4,] NA NA NA NA
> [5,] NA NA NA NA
?matrix
Usage
matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
dimnames = NULL)
so we don't even need preallocate_matrix() function
> x < .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> x
[,1] [,2] [,3] [,4]
20190906 17:07:27 NA NA NA NA
20190906 17:07:28 NA NA NA NA
20190906 17:07:29 NA NA NA NA
20190906 17:07:30 NA NA NA NA
20190906 17:07:31 NA NA NA NA
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Hi Daniel
Thanks a lot.
Those are very helpful ideas.
rbind_append > it still has to allocate memory for the resulting
series... so if memory allocation was the main reason for slow performance,
maybe rbind_append doesn't change much? what do you think?
preallocating regularinterval timeseries is a good idea.
however financial data are irregularly spaced (sometimes there may not be
any price updates for a few secs, even more).
even if we postulate that prices are allowed to change no more than once
per second, there's a lot of uses for the frequency of price updates, not
only the values of the prices (the simplest assumption is the poisson
arrival process for the updates, but there are many fancier, more powerful
models...)
so, preallocating a regularly spaced 1sec interval xts series dumbs down
many things!
i wish i could preallocate the vector for the values and maybe indices,
but then do the assignment of the sort:
(say, in C++ i would have a method)
price.set_next_point(time, value);
thanks!
On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka < [hidden email]>
wrote:
>
>
> > Wiadomość napisana przez Daniel Cegiełka < [hidden email]> w
> dniu 06.09.2019, o godz. 16:10:
> >
>
> >
> > 2) preallocation
> >
> > preallocate_matrix < function(n)
> > {
> > x < matrix()
> > length(x) < 4 * n # bid, ask, bid_size, ask_size
> > dim(x) < c(n, 4) # see: ?dim
> > return(x)
> > }
> >
> > > x < preallocate_matrix(5)
> > > x
> > [,1] [,2] [,3] [,4]
> > [1,] NA NA NA NA
> > [2,] NA NA NA NA
> > [3,] NA NA NA NA
> > [4,] NA NA NA NA
> > [5,] NA NA NA NA
>
> ?matrix
>
> Usage
> matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
> dimnames = NULL)
>
> so we don't even need preallocate_matrix() function
>
> > x < .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> > x
> [,1] [,2] [,3] [,4]
> 20190906 17:07:27 NA NA NA NA
> 20190906 17:07:28 NA NA NA NA
> 20190906 17:07:29 NA NA NA NA
> 20190906 17:07:30 NA NA NA NA
> 20190906 17:07:31 NA NA NA NA
>
>
>
>
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Vladimir,
Questiondo you have to do operations on the newly updated xts each time
you append it? For instance, when I backtest strategies and I don't know
how long the output would be, I start off by allocating an empty list,
keep updating the list, and rbind it at the end.
EG:
results < list()
for(i in 1:n){
#output < do something
results[[i]] <output
}
results < do.call(rbind, results)
This might help, as this doesn't keep copying the larger list over and over
again, which I remember doing before I learned about this method.
Hope this helps.
Ilya
On Fri, Sep 6, 2019 at 2:05 PM Vladimir Morozov < [hidden email]>
wrote:
> Hi Daniel
> Thanks a lot.
> Those are very helpful ideas.
>
> rbind_append > it still has to allocate memory for the resulting
> series... so if memory allocation was the main reason for slow performance,
> maybe rbind_append doesn't change much? what do you think?
>
> preallocating regularinterval timeseries is a good idea.
> however financial data are irregularly spaced (sometimes there may not be
> any price updates for a few secs, even more).
> even if we postulate that prices are allowed to change no more than once
> per second, there's a lot of uses for the frequency of price updates, not
> only the values of the prices (the simplest assumption is the poisson
> arrival process for the updates, but there are many fancier, more powerful
> models...)
> so, preallocating a regularly spaced 1sec interval xts series dumbs down
> many things!
>
> i wish i could preallocate the vector for the values and maybe indices,
> but then do the assignment of the sort:
> (say, in C++ i would have a method)
> price.set_next_point(time, value);
>
> thanks!
>
> On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka < [hidden email]
> >
> wrote:
>
> >
> >
> > > Wiadomość napisana przez Daniel Cegiełka < [hidden email]> w
> > dniu 06.09.2019, o godz. 16:10:
> > >
> >
> > >
> > > 2) preallocation
> > >
> > > preallocate_matrix < function(n)
> > > {
> > > x < matrix()
> > > length(x) < 4 * n # bid, ask, bid_size, ask_size
> > > dim(x) < c(n, 4) # see: ?dim
> > > return(x)
> > > }
> > >
> > > > x < preallocate_matrix(5)
> > > > x
> > > [,1] [,2] [,3] [,4]
> > > [1,] NA NA NA NA
> > > [2,] NA NA NA NA
> > > [3,] NA NA NA NA
> > > [4,] NA NA NA NA
> > > [5,] NA NA NA NA
> >
> > ?matrix
> >
> > Usage
> > matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
> > dimnames = NULL)
> >
> > so we don't even need preallocate_matrix() function
> >
> > > x < .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> > > x
> > [,1] [,2] [,3] [,4]
> > 20190906 17:07:27 NA NA NA NA
> > 20190906 17:07:28 NA NA NA NA
> > 20190906 17:07:29 NA NA NA NA
> > 20190906 17:07:30 NA NA NA NA
> > 20190906 17:07:31 NA NA NA NA
> >
> >
> >
> >
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions
> should go.
>
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Wiadomość napisana przez Vladimir Morozov < [hidden email]> w dniu 06.09.2019, o godz. 20:04:
Hi Daniel Thanks a lot. Those are very helpful ideas.
rbind_append > it still has to allocate memory for the resulting series... so if memory allocation was the main reason for slow performance, maybe rbind_append doesn't change much? what do you think?
preallocating regularinterval timeseries is a good idea. however financial data are irregularly spaced (sometimes there may not be any price updates for a few secs, even more). even if we postulate that prices are allowed to change no more than once per second, there's a lot of uses for the frequency of price updates, not only the values of the prices (the simplest assumption is the poisson arrival process for the updates, but there are many fancier, more powerful models...) so, preallocating a regularly spaced 1sec interval xts series dumbs down many things!
let's start with what exactly do you want to do? Do you want to collect market data and save it to disk? Or maybe you want to have a realtime strategy? These are two different problems and require distinct solutions.
1) market data storage  why do you want to use R here? Isn't it better to dump the memory using mmap syscall and then import it into the database or R?
2) realtime market strategy in R  in this case your lookback is limited. So if you add new data point, you can also discard/drop the oldest. In this way, your memory usage will remain at the same low level. If this solution suits you, then you can write a fast function in C here that would operate on the xts object.
There is no such thing as matrix in R  this is a multidimensional vector. Let's say we have classic OHLC data for xts object:
O H L C O H L C O H L C O H L C O H L C
In the memory of the data looks like one long vector.
x: OOOOOHHHHHLLLLLCCCCC
You can be clever here and use memcpy():
memcpy(&xp + 1, &xp, (nrows(x)  1)) * sizeof(double)); // or int  use: switch((TYPEOF(x))
memcpy(&index_p + 1, &index_p, (nrows(x)  1)) * sizeof(double)); // or int for Date() type
This will move the memory so that the oldest value will be overwritten:
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 x: OOOOH HHHHL LLLLC CCCC N
Then you can add a new index and value.
You will have preallocated memory at all times and you will use memory copy as little as possible. And the most important: you'll be operating on the xts object all time, so your code in R will be very fast :)
It is advanced solutions  you need to understand not only how R's internals works, but also have a good C skills. If you want to use R for realtime trading, it's worth learn these things.
Daniel
i wish i could preallocate the vector for the values and maybe indices, but then do the assignment of the sort: (say, in C++ i would have a method) price.set_next_point(time, value);
thanks! On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka < [hidden email]> wrote:
> Wiadomość napisana przez Daniel Cegiełka <[hidden email]> w dniu 06.09.2019, o godz. 16:10:
>
>
> 2) preallocation
>
> preallocate_matrix < function(n)
> {
> x < matrix()
> length(x) < 4 * n # bid, ask, bid_size, ask_size
> dim(x) < c(n, 4) # see: ?dim
> return(x)
> }
>
> > x < preallocate_matrix(5)
> > x
> [,1] [,2] [,3] [,4]
> [1,] NA NA NA NA
> [2,] NA NA NA NA
> [3,] NA NA NA NA
> [4,] NA NA NA NA
> [5,] NA NA NA NA
?matrix
Usage
matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
dimnames = NULL)
so we don't even need preallocate_matrix() function
> x < .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> x
[,1] [,2] [,3] [,4]
20190906 17:07:27 NA NA NA NA
20190906 17:07:28 NA NA NA NA
20190906 17:07:29 NA NA NA NA
20190906 17:07:30 NA NA NA NA
20190906 17:07:31 NA NA NA NA
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Daniel
It's a brilliant idea
I'll try it out
Yes, it's quite advanced
On Sat, Sep 7, 2019, 4:07 AM Daniel Cegiełka < [hidden email]>
wrote:
>
>
> Wiadomość napisana przez Vladimir Morozov < [hidden email]> w dniu
> 06.09.2019, o godz. 20:04:
>
> Hi Daniel
> Thanks a lot.
> Those are very helpful ideas.
>
> rbind_append > it still has to allocate memory for the resulting
> series... so if memory allocation was the main reason for slow performance,
> maybe rbind_append doesn't change much? what do you think?
>
> preallocating regularinterval timeseries is a good idea.
> however financial data are irregularly spaced (sometimes there may not be
> any price updates for a few secs, even more).
> even if we postulate that prices are allowed to change no more than once
> per second, there's a lot of uses for the frequency of price updates, not
> only the values of the prices (the simplest assumption is the poisson
> arrival process for the updates, but there are many fancier, more powerful
> models...)
> so, preallocating a regularly spaced 1sec interval xts series dumbs down
> many things!
>
>
> let's start with what exactly do you want to do? Do you want to collect
> market data and save it to disk? Or maybe you want to have a realtime
> strategy? These are two different problems and require distinct solutions.
>
> 1) market data storage  why do you want to use R here? Isn't it better to
> dump the memory using mmap syscall and then import it into the database or
> R?
>
> 2) realtime market strategy in R  in this case your lookback is limited.
> So if you add new data point, you can also discard/drop the oldest. In this
> way, your memory usage will remain at the same low level. If this solution
> suits you, then you can write a fast function in C here that would operate
> on the xts object.
>
> There is no such thing as matrix in R  this is a multidimensional vector.
> Let's say we have classic OHLC data for xts object:
>
> O H L C
> O H L C
> O H L C
> O H L C
> O H L C
>
>
> In the memory of the data looks like one long vector.
>
> x: OOOOOHHHHHLLLLLCCCCC
>
> You can be clever here and use memcpy():
>
> memcpy(&xp + 1, &xp, (nrows(x)  1)) * sizeof(double)); // or int  use:
> switch((TYPEOF(x))
>
> memcpy(&index_p + 1, &index_p, (nrows(x)  1)) * sizeof(double)); // or
> int for Date() type
>
> This will move the memory so that the oldest value will be overwritten:
>
> 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
> x: OOOOH HHHHL LLLLC CCCC N
>
> Then you can add a new index and value.
>
> You will have preallocated memory at all times and you will use memory
> copy as little as possible. And the most important: you'll be operating on
> the xts object all time, so your code in R will be very fast :)
>
> It is advanced solutions  you need to understand not only how R's
> internals works, but also have a good C skills. If you want to use R for
> realtime trading, it's worth learn these things.
>
> Daniel
>
>
>
>
>
> i wish i could preallocate the vector for the values and maybe indices,
> but then do the assignment of the sort:
> (say, in C++ i would have a method)
> price.set_next_point(time, value);
>
> thanks!
>
> On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka < [hidden email]>
> wrote:
>
>>
>>
>> > Wiadomość napisana przez Daniel Cegiełka < [hidden email]> w
>> dniu 06.09.2019, o godz. 16:10:
>> >
>>
>> >
>> > 2) preallocation
>> >
>> > preallocate_matrix < function(n)
>> > {
>> > x < matrix()
>> > length(x) < 4 * n # bid, ask, bid_size, ask_size
>> > dim(x) < c(n, 4) # see: ?dim
>> > return(x)
>> > }
>> >
>> > > x < preallocate_matrix(5)
>> > > x
>> > [,1] [,2] [,3] [,4]
>> > [1,] NA NA NA NA
>> > [2,] NA NA NA NA
>> > [3,] NA NA NA NA
>> > [4,] NA NA NA NA
>> > [5,] NA NA NA NA
>>
>> ?matrix
>>
>> Usage
>> matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
>> dimnames = NULL)
>>
>> so we don't even need preallocate_matrix() function
>>
>> > x < .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
>> > x
>> [,1] [,2] [,3] [,4]
>> 20190906 17:07:27 NA NA NA NA
>> 20190906 17:07:28 NA NA NA NA
>> 20190906 17:07:29 NA NA NA NA
>> 20190906 17:07:30 NA NA NA NA
>> 20190906 17:07:31 NA NA NA NA
>>
>>
>>
>>
>
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Thanks, Ilya
On Sat, Sep 7, 2019, 3:47 AM Ilya Kipnis < [hidden email]> wrote:
> Vladimir,
>
> Questiondo you have to do operations on the newly updated xts each time
> you append it? For instance, when I backtest strategies and I don't know
> how long the output would be, I start off by allocating an empty list,
> keep updating the list, and rbind it at the end.
>
> EG:
>
> results < list()
> for(i in 1:n){
> #output < do something
> results[[i]] <output
> }
> results < do.call(rbind, results)
>
> This might help, as this doesn't keep copying the larger list over and
> over again, which I remember doing before I learned about this method.
>
> Hope this helps.
>
> Ilya
>
> On Fri, Sep 6, 2019 at 2:05 PM Vladimir Morozov < [hidden email]>
> wrote:
>
>> Hi Daniel
>> Thanks a lot.
>> Those are very helpful ideas.
>>
>> rbind_append > it still has to allocate memory for the resulting
>> series... so if memory allocation was the main reason for slow
>> performance,
>> maybe rbind_append doesn't change much? what do you think?
>>
>> preallocating regularinterval timeseries is a good idea.
>> however financial data are irregularly spaced (sometimes there may not be
>> any price updates for a few secs, even more).
>> even if we postulate that prices are allowed to change no more than once
>> per second, there's a lot of uses for the frequency of price updates, not
>> only the values of the prices (the simplest assumption is the poisson
>> arrival process for the updates, but there are many fancier, more powerful
>> models...)
>> so, preallocating a regularly spaced 1sec interval xts series dumbs down
>> many things!
>>
>> i wish i could preallocate the vector for the values and maybe indices,
>> but then do the assignment of the sort:
>> (say, in C++ i would have a method)
>> price.set_next_point(time, value);
>>
>> thanks!
>>
>> On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <
>> [hidden email]>
>> wrote:
>>
>> >
>> >
>> > > Wiadomość napisana przez Daniel Cegiełka < [hidden email]>
>> w
>> > dniu 06.09.2019, o godz. 16:10:
>> > >
>> >
>> > >
>> > > 2) preallocation
>> > >
>> > > preallocate_matrix < function(n)
>> > > {
>> > > x < matrix()
>> > > length(x) < 4 * n # bid, ask, bid_size, ask_size
>> > > dim(x) < c(n, 4) # see: ?dim
>> > > return(x)
>> > > }
>> > >
>> > > > x < preallocate_matrix(5)
>> > > > x
>> > > [,1] [,2] [,3] [,4]
>> > > [1,] NA NA NA NA
>> > > [2,] NA NA NA NA
>> > > [3,] NA NA NA NA
>> > > [4,] NA NA NA NA
>> > > [5,] NA NA NA NA
>> >
>> > ?matrix
>> >
>> > Usage
>> > matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
>> > dimnames = NULL)
>> >
>> > so we don't even need preallocate_matrix() function
>> >
>> > > x < .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
>> > > x
>> > [,1] [,2] [,3] [,4]
>> > 20190906 17:07:27 NA NA NA NA
>> > 20190906 17:07:28 NA NA NA NA
>> > 20190906 17:07:29 NA NA NA NA
>> > 20190906 17:07:30 NA NA NA NA
>> > 20190906 17:07:31 NA NA NA NA
>> >
>> >
>> >
>> >
>>
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>  Subscriberposting only. If you want to post, subscribe first.
>>  Also note that this is not the rhelp list where general R questions
>> should go.
>>
>
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.

