how to grow XTS series in R dynamically ? And Quickly!

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

how to grow XTS series in R dynamically ? And Quickly!

Vladimir Morozov-2
Dear xts experts

I use R and XTS to store dynamically growing price time-series for currency
pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing with each new
incoming market price - say, one per second).

I want to speed up my R function that generates and rbinds a few new rows
to multiple Global XTS objects. This function us called often (sometimes
once per second) and takes a lot of time.

Specifically, I have a function add.newRate.EUR(POSIXct Time, float Rate)
This function performs some time-consuming manipulations, then modifies
(global objects) XTS series1 through XTS seriesN. Specifically, I rbind
some new rows to each of series1 through seriesN in global environment.
(specific example - a new Rate for EUR/USD comes into the function. This
function needs to rbind new point to EUR.USD xts object, but it also needs
to calculate new value for cross-currency pairs, like EUR/JPY, EUR/GPB,
EUR/CHF, EUR/CAD, EUR/AUD, EUR/NZD, and rbind new point to those XTS series
as well)

it all happens quite slowly in R. How do I accelerate it?

To avoid copying XTS series between the function and the global
environment, the function attempts to modify the seriesX in the global
environment, and not return XTS on function return.
The structure of my project is such that my other tasks in R need to access
these XTS series1 through seriesN in Global Environment.
Still, this whole set-up is rather slow.

Do you think using Rcpp to move cross-currency rate calculations to C++
could help?
E.g. I want to write add.newRate.EUR.RCPP(int Time, float Rate) in C++. I
think it will perform internal manipulations of time T and value X a little
faster.

Second idea for speed up:  doing rbind in C++ using package RcppXts.
I plan to use the following function in RcppXts package:
    function("xtsRbind",
             &xtsRbind,
             List::create(Named("x"), Named("y"), Named("dup")),
             "Combine two xts objects row-wise");
Then I want to use Rcpp function assign( name, x ) to assign the created
object back to GlobalEnvironment.

Do you think the above way is a good way to grow dynamical XTS series inside
Rcpp?
Do you think using Rcpp and RcppXts will provide significant acceleration
compared to pure R ?

Thank you
Vlad

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Vladimir Morozov-2
did some googling on my question
general wisdom on the internet seems to be - it's SLOW to grow R lists in a
loop. Must Pre-allocate First.

Question: is there a feature in XTS that allows to pre-allocate space for
growing XTS series?
thanks again!

On Fri, Sep 6, 2019 at 11:51 AM Vladimir Morozov <[hidden email]>
wrote:

> Dear xts experts
>
> I use R and XTS to store dynamically growing price time-series for
> currency pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing with
> each new incoming market price - say, one per second).
>
> I want to speed up my R function that generates and rbinds a few new rows
> to multiple Global XTS objects. This function us called often (sometimes
> once per second) and takes a lot of time.
>
> Specifically, I have a function add.newRate.EUR(POSIXct Time, float Rate)
> This function performs some time-consuming manipulations, then modifies
> (global objects) XTS series1 through XTS seriesN. Specifically, I rbind
> some new rows to each of series1 through seriesN in global environment.
> (specific example - a new Rate for EUR/USD comes into the function. This
> function needs to rbind new point to EUR.USD xts object, but it also needs
> to calculate new value for cross-currency pairs, like EUR/JPY, EUR/GPB,
> EUR/CHF, EUR/CAD, EUR/AUD, EUR/NZD, and rbind new point to those XTS series
> as well)
>
> it all happens quite slowly in R. How do I accelerate it?
>
> To avoid copying XTS series between the function and the global
> environment, the function attempts to modify the seriesX in the global
> environment, and not return XTS on function return.
> The structure of my project is such that my other tasks in R need to
> access these XTS series1 through seriesN in Global Environment.
> Still, this whole set-up is rather slow.
>
> Do you think using Rcpp to move cross-currency rate calculations to C++
> could help?
> E.g. I want to write add.newRate.EUR.RCPP(int Time, float Rate) in C++. I
> think it will perform internal manipulations of time T and value X a little
> faster.
>
> Second idea for speed up:  doing rbind in C++ using package RcppXts.
> I plan to use the following function in RcppXts package:
>     function("xtsRbind",
>              &xtsRbind,
>              List::create(Named("x"), Named("y"), Named("dup")),
>              "Combine two xts objects row-wise");
> Then I want to use Rcpp function assign( name, x ) to assign the created
> object back to GlobalEnvironment.
>
> Do you think the above way is a good way to grow dynamical XTS series inside
> Rcpp?
> Do you think using Rcpp and RcppXts will provide significant acceleration
> compared to pure R ?
>
> Thank you
> Vlad
>
>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

mark leeds
Hi Vladimir: I can tell that  you tried to explain your problem clearly but
you'd get more
replies if you actually showed the R code that you wrote. Minimal,
reproducible examples
get a lot more responses.  Good luck.
                                                                      Mark

On Fri, Sep 6, 2019 at 12:05 AM Vladimir Morozov <[hidden email]>
wrote:

> did some googling on my question
> general wisdom on the internet seems to be - it's SLOW to grow R lists in a
> loop. Must Pre-allocate First.
>
> Question: is there a feature in XTS that allows to pre-allocate space for
> growing XTS series?
> thanks again!
>
> On Fri, Sep 6, 2019 at 11:51 AM Vladimir Morozov <[hidden email]>
> wrote:
>
> > Dear xts experts
> >
> > I use R and XTS to store dynamically growing price time-series for
> > currency pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing
> with
> > each new incoming market price - say, one per second).
> >
> > I want to speed up my R function that generates and rbinds a few new rows
> > to multiple Global XTS objects. This function us called often (sometimes
> > once per second) and takes a lot of time.
> >
> > Specifically, I have a function add.newRate.EUR(POSIXct Time, float Rate)
> > This function performs some time-consuming manipulations, then modifies
> > (global objects) XTS series1 through XTS seriesN. Specifically, I rbind
> > some new rows to each of series1 through seriesN in global environment.
> > (specific example - a new Rate for EUR/USD comes into the function. This
> > function needs to rbind new point to EUR.USD xts object, but it also
> needs
> > to calculate new value for cross-currency pairs, like EUR/JPY, EUR/GPB,
> > EUR/CHF, EUR/CAD, EUR/AUD, EUR/NZD, and rbind new point to those XTS
> series
> > as well)
> >
> > it all happens quite slowly in R. How do I accelerate it?
> >
> > To avoid copying XTS series between the function and the global
> > environment, the function attempts to modify the seriesX in the global
> > environment, and not return XTS on function return.
> > The structure of my project is such that my other tasks in R need to
> > access these XTS series1 through seriesN in Global Environment.
> > Still, this whole set-up is rather slow.
> >
> > Do you think using Rcpp to move cross-currency rate calculations to C++
> > could help?
> > E.g. I want to write add.newRate.EUR.RCPP(int Time, float Rate) in C++. I
> > think it will perform internal manipulations of time T and value X a
> little
> > faster.
> >
> > Second idea for speed up:  doing rbind in C++ using package RcppXts.
> > I plan to use the following function in RcppXts package:
> >     function("xtsRbind",
> >              &xtsRbind,
> >              List::create(Named("x"), Named("y"), Named("dup")),
> >              "Combine two xts objects row-wise");
> > Then I want to use Rcpp function assign( name, x ) to assign the created
> > object back to GlobalEnvironment.
> >
> > Do you think the above way is a good way to grow dynamical XTS series
> inside
> > Rcpp?
> > Do you think using Rcpp and RcppXts will provide significant acceleration
> > compared to pure R ?
> >
> > Thank you
> > Vlad
> >
> >
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Daniel Cegiełka
In reply to this post by Vladimir Morozov-2


Wiadomość napisana przez Vladimir Morozov <[hidden email]> w dniu 06.09.2019, o godz. 04:51:

Dear xts experts

I use R and XTS to store dynamically growing price time-series for currency
pair rates (e.g. time series of EUR/USD, EUR/JPY, etc growing with each new
incoming market price - say, one per second).



(…)

it all happens quite slowly in R. How do I accelerate it?


1) forgotten xts acceleration:


to activate it add the -DRBIND_APPEND flag to Makevars (linux/unix/macos) or Makevars.win (Windows), and recompile xts sources.



2) preallocation

preallocate_matrix <- function(n)
{
    x <- matrix()
    length(x) <- 4 * n      # bid, ask, bid_size, ask_size
    dim(x) <- c(n, 4)       # see: ?dim
    return(x)
}

> x <- preallocate_matrix(5)
> x
     [,1] [,2] [,3] [,4]
[1,]   NA   NA   NA   NA
[2,]   NA   NA   NA   NA
[3,]   NA   NA   NA   NA
[4,]   NA   NA   NA   NA
[5,]   NA   NA   NA   NA

Now you need xts index, eg:

i <- Sys.time() + 1:5
> i
[1] "2019-09-06 15:47:48 CEST" "2019-09-06 15:47:49 CEST" "2019-09-06 15:47:50 CEST" "2019-09-06 15:47:51 CEST"
[5] "2019-09-06 15:47:52 CEST”


Our xts object:

> x <- .xts(x, index = i)
                    [,1] [,2] [,3] [,4]
2019-09-06 15:47:48   NA   NA   NA   NA
2019-09-06 15:47:49   NA   NA   NA   NA
2019-09-06 15:47:50   NA   NA   NA   NA
2019-09-06 15:47:51   NA   NA   NA   NA
2019-09-06 15:47:52   NA   NA   NA   NA


And now you can use this xts object without memory copying:


> x[1,] <- c(1, 2, 3, 4)
> x
                    [,1] [,2] [,3] [,4]
2019-09-06 15:47:48    1    2    3    4
2019-09-06 15:47:49   NA   NA   NA   NA
2019-09-06 15:47:50   NA   NA   NA   NA
2019-09-06 15:47:51   NA   NA   NA   NA
2019-09-06 15:47:52   NA   NA   NA   NA

> x[2,] <- c(11, 22, 33, 44)
> x
                    [,1] [,2] [,3] [,4]
2019-09-06 15:47:48    1    2    3    4
2019-09-06 15:47:49   11   22   33   44
2019-09-06 15:47:50   NA   NA   NA   NA
2019-09-06 15:47:51   NA   NA   NA   NA
2019-09-06 15:47:52   NA   NA   NA   NA


etc.

That's what you meant.

Best,
Daniel


_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Daniel Cegiełka


> Wiadomość napisana przez Daniel Cegiełka <[hidden email]> w dniu 06.09.2019, o godz. 16:10:
>

>
> 2) preallocation
>
> preallocate_matrix <- function(n)
> {
>     x <- matrix()
>     length(x) <- 4 * n      # bid, ask, bid_size, ask_size
>     dim(x) <- c(n, 4)       # see: ?dim
>     return(x)
> }
>
> > x <- preallocate_matrix(5)
> > x
>      [,1] [,2] [,3] [,4]
> [1,]   NA   NA   NA   NA
> [2,]   NA   NA   NA   NA
> [3,]   NA   NA   NA   NA
> [4,]   NA   NA   NA   NA
> [5,]   NA   NA   NA   NA
?matrix

Usage
matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
       dimnames = NULL)

so we don't even need preallocate_matrix() function

> x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> x
                    [,1] [,2] [,3] [,4]
2019-09-06 17:07:27   NA   NA   NA   NA
2019-09-06 17:07:28   NA   NA   NA   NA
2019-09-06 17:07:29   NA   NA   NA   NA
2019-09-06 17:07:30   NA   NA   NA   NA
2019-09-06 17:07:31   NA   NA   NA   NA




_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Vladimir Morozov-2
Hi Daniel
Thanks a lot.
Those are very helpful ideas.

rbind_append --> it still has to allocate memory for the resulting
series... so if memory allocation was the main reason for slow performance,
maybe rbind_append doesn't change much? what do you think?

preallocating regular-interval time-series is a good idea.
however financial data are irregularly spaced (sometimes there may not be
any price updates for a few secs, even more).
even if we postulate that prices are allowed to change no more than once
per second, there's a lot of uses for the frequency of price updates, not
only the values of the prices (the simplest assumption is the poisson
arrival process for the updates, but there are many fancier, more powerful
models...)
so, pre-allocating a regularly spaced 1-sec interval xts series dumbs down
many things!

i wish i could pre-allocate the vector for the values and maybe indices,
but then do the assignment of the sort:
(say, in C++ i would have a method)
    price.set_next_point(time, value);

thanks!

On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <[hidden email]>
wrote:

>
>
> > Wiadomość napisana przez Daniel Cegiełka <[hidden email]> w
> dniu 06.09.2019, o godz. 16:10:
> >
>
> >
> > 2) preallocation
> >
> > preallocate_matrix <- function(n)
> > {
> >     x <- matrix()
> >     length(x) <- 4 * n      # bid, ask, bid_size, ask_size
> >     dim(x) <- c(n, 4)       # see: ?dim
> >     return(x)
> > }
> >
> > > x <- preallocate_matrix(5)
> > > x
> >      [,1] [,2] [,3] [,4]
> > [1,]   NA   NA   NA   NA
> > [2,]   NA   NA   NA   NA
> > [3,]   NA   NA   NA   NA
> > [4,]   NA   NA   NA   NA
> > [5,]   NA   NA   NA   NA
>
> ?matrix
>
> Usage
> matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
>        dimnames = NULL)
>
> so we don't even need preallocate_matrix() function
>
> > x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> > x
>                     [,1] [,2] [,3] [,4]
> 2019-09-06 17:07:27   NA   NA   NA   NA
> 2019-09-06 17:07:28   NA   NA   NA   NA
> 2019-09-06 17:07:29   NA   NA   NA   NA
> 2019-09-06 17:07:30   NA   NA   NA   NA
> 2019-09-06 17:07:31   NA   NA   NA   NA
>
>
>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Ilya Kipnis
Vladimir,

Question--do you have to do operations on the newly updated xts each time
you append it? For instance, when I backtest strategies and I don't know
how long the output would be, I start off by allocating an empty list,
keep updating the list, and rbind it at the end.

EG:

results <- list()
for(i in 1:n){
  #output <- do something
  results[[i]] <-output
}
results <- do.call(rbind, results)

This might help, as this doesn't keep copying the larger list over and over
again, which I remember doing before I learned about this method.

Hope this helps.

-Ilya

On Fri, Sep 6, 2019 at 2:05 PM Vladimir Morozov <[hidden email]>
wrote:

> Hi Daniel
> Thanks a lot.
> Those are very helpful ideas.
>
> rbind_append --> it still has to allocate memory for the resulting
> series... so if memory allocation was the main reason for slow performance,
> maybe rbind_append doesn't change much? what do you think?
>
> preallocating regular-interval time-series is a good idea.
> however financial data are irregularly spaced (sometimes there may not be
> any price updates for a few secs, even more).
> even if we postulate that prices are allowed to change no more than once
> per second, there's a lot of uses for the frequency of price updates, not
> only the values of the prices (the simplest assumption is the poisson
> arrival process for the updates, but there are many fancier, more powerful
> models...)
> so, pre-allocating a regularly spaced 1-sec interval xts series dumbs down
> many things!
>
> i wish i could pre-allocate the vector for the values and maybe indices,
> but then do the assignment of the sort:
> (say, in C++ i would have a method)
>     price.set_next_point(time, value);
>
> thanks!
>
> On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <[hidden email]
> >
> wrote:
>
> >
> >
> > > Wiadomość napisana przez Daniel Cegiełka <[hidden email]> w
> > dniu 06.09.2019, o godz. 16:10:
> > >
> >
> > >
> > > 2) preallocation
> > >
> > > preallocate_matrix <- function(n)
> > > {
> > >     x <- matrix()
> > >     length(x) <- 4 * n      # bid, ask, bid_size, ask_size
> > >     dim(x) <- c(n, 4)       # see: ?dim
> > >     return(x)
> > > }
> > >
> > > > x <- preallocate_matrix(5)
> > > > x
> > >      [,1] [,2] [,3] [,4]
> > > [1,]   NA   NA   NA   NA
> > > [2,]   NA   NA   NA   NA
> > > [3,]   NA   NA   NA   NA
> > > [4,]   NA   NA   NA   NA
> > > [5,]   NA   NA   NA   NA
> >
> > ?matrix
> >
> > Usage
> > matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
> >        dimnames = NULL)
> >
> > so we don't even need preallocate_matrix() function
> >
> > > x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> > > x
> >                     [,1] [,2] [,3] [,4]
> > 2019-09-06 17:07:27   NA   NA   NA   NA
> > 2019-09-06 17:07:28   NA   NA   NA   NA
> > 2019-09-06 17:07:29   NA   NA   NA   NA
> > 2019-09-06 17:07:30   NA   NA   NA   NA
> > 2019-09-06 17:07:31   NA   NA   NA   NA
> >
> >
> >
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Daniel Cegiełka
In reply to this post by Vladimir Morozov-2


Wiadomość napisana przez Vladimir Morozov <[hidden email]> w dniu 06.09.2019, o godz. 20:04:

Hi Daniel
Thanks a lot.
Those are very helpful ideas.

rbind_append --> it still has to allocate memory for the resulting series... so if memory allocation was the main reason for slow performance, maybe rbind_append doesn't change much? what do you think?

preallocating regular-interval time-series is a good idea.
however financial data are irregularly spaced (sometimes there may not be any price updates for a few secs, even more).
even if we postulate that prices are allowed to change no more than once per second, there's a lot of uses for the frequency of price updates, not only the values of the prices (the simplest assumption is the poisson arrival process for the updates, but there are many fancier, more powerful models...)
so, pre-allocating a regularly spaced 1-sec interval xts series dumbs down many things!


let's start with what exactly do you want to do? Do you want to collect market data and save it to disk? Or maybe you want to have a real-time strategy? These are two different problems and require distinct solutions.

1) market data storage - why do you want to use R here? Isn't it better to dump the memory using mmap syscall and then import it into the database or R?

2) real-time market strategy in R - in this case your lookback is limited. So if you add new data point, you can also discard/drop the oldest. In this way, your memory usage will remain at the same low level. If this solution suits you, then you can write a fast function in C here that would operate on the xts object.

There is no such thing as matrix in R - this is a multidimensional vector. Let's say we have classic OHLC data for xts object:

O H L C
O H L C
O H L C
O H L C
O H L C


In the memory of the data looks like one long vector.

x: OOOOOHHHHHLLLLLCCCCC

You can be clever here and use memcpy():

memcpy(&xp + 1, &xp, (nrows(x) - 1)) * sizeof(double));  // or int - use: switch((TYPEOF(x))

memcpy(&index_p + 1, &index_p, (nrows(x) - 1)) * sizeof(double)); // or int for Date() type

This will move the memory so that the oldest value will be overwritten:

    1 2 3 4 5   1 2 3 4 5   1 2 3 4 5   1 2 3 4 5
x: OOOOH   HHHHL      LLLLC      CCCC N

Then you can add a new index and value.

You will have preallocated memory at all times and you will use memory copy as little as possible. And the most important: you'll be operating on the xts object all time, so your code in R will be very fast :)

It is advanced solutions - you need to understand not only how R's internals works, but also have a good C skills. If you want to use R for real-time trading, it's worth learn these things.

Daniel





i wish i could pre-allocate the vector for the values and maybe indices, but then do the assignment of the sort:
(say, in C++ i would have a method)
    price.set_next_point(time, value);

thanks!

On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <[hidden email]> wrote:


> Wiadomość napisana przez Daniel Cegiełka <[hidden email]> w dniu 06.09.2019, o godz. 16:10:
>

>
> 2) preallocation
>
> preallocate_matrix <- function(n)
> {
>     x <- matrix()
>     length(x) <- 4 * n      # bid, ask, bid_size, ask_size
>     dim(x) <- c(n, 4)       # see: ?dim
>     return(x)
> }
>
> > x <- preallocate_matrix(5)
> > x
>      [,1] [,2] [,3] [,4]
> [1,]   NA   NA   NA   NA
> [2,]   NA   NA   NA   NA
> [3,]   NA   NA   NA   NA
> [4,]   NA   NA   NA   NA
> [5,]   NA   NA   NA   NA

?matrix

Usage
matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
       dimnames = NULL)

so we don't even need preallocate_matrix() function

> x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
> x
                    [,1] [,2] [,3] [,4]
2019-09-06 17:07:27   NA   NA   NA   NA
2019-09-06 17:07:28   NA   NA   NA   NA
2019-09-06 17:07:29   NA   NA   NA   NA
2019-09-06 17:07:30   NA   NA   NA   NA
2019-09-06 17:07:31   NA   NA   NA   NA





_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Vladimir Morozov-2
Daniel
It's a brilliant idea
I'll try it out
Yes, it's quite advanced

On Sat, Sep 7, 2019, 4:07 AM Daniel Cegiełka <[hidden email]>
wrote:

>
>
> Wiadomość napisana przez Vladimir Morozov <[hidden email]> w dniu
> 06.09.2019, o godz. 20:04:
>
> Hi Daniel
> Thanks a lot.
> Those are very helpful ideas.
>
> rbind_append --> it still has to allocate memory for the resulting
> series... so if memory allocation was the main reason for slow performance,
> maybe rbind_append doesn't change much? what do you think?
>
> preallocating regular-interval time-series is a good idea.
> however financial data are irregularly spaced (sometimes there may not be
> any price updates for a few secs, even more).
> even if we postulate that prices are allowed to change no more than once
> per second, there's a lot of uses for the frequency of price updates, not
> only the values of the prices (the simplest assumption is the poisson
> arrival process for the updates, but there are many fancier, more powerful
> models...)
> so, pre-allocating a regularly spaced 1-sec interval xts series dumbs down
> many things!
>
>
> let's start with what exactly do you want to do? Do you want to collect
> market data and save it to disk? Or maybe you want to have a real-time
> strategy? These are two different problems and require distinct solutions.
>
> 1) market data storage - why do you want to use R here? Isn't it better to
> dump the memory using mmap syscall and then import it into the database or
> R?
>
> 2) real-time market strategy in R - in this case your lookback is limited.
> So if you add new data point, you can also discard/drop the oldest. In this
> way, your memory usage will remain at the same low level. If this solution
> suits you, then you can write a fast function in C here that would operate
> on the xts object.
>
> There is no such thing as matrix in R - this is a multidimensional vector.
> Let's say we have classic OHLC data for xts object:
>
> O H L C
> O H L C
> O H L C
> O H L C
> O H L C
>
>
> In the memory of the data looks like one long vector.
>
> x: OOOOOHHHHHLLLLLCCCCC
>
> You can be clever here and use memcpy():
>
> memcpy(&xp + 1, &xp, (nrows(x) - 1)) * sizeof(double));  // or int - use:
> switch((TYPEOF(x))
>
> memcpy(&index_p + 1, &index_p, (nrows(x) - 1)) * sizeof(double)); // or
> int for Date() type
>
> This will move the memory so that the oldest value will be overwritten:
>
>     1 2 3 4 5   1 2 3 4 5   1 2 3 4 5   1 2 3 4 5
> x: OOOOH   HHHHL      LLLLC      CCCC N
>
> Then you can add a new index and value.
>
> You will have preallocated memory at all times and you will use memory
> copy as little as possible. And the most important: you'll be operating on
> the xts object all time, so your code in R will be very fast :)
>
> It is advanced solutions - you need to understand not only how R's
> internals works, but also have a good C skills. If you want to use R for
> real-time trading, it's worth learn these things.
>
> Daniel
>
>
>
>
>
> i wish i could pre-allocate the vector for the values and maybe indices,
> but then do the assignment of the sort:
> (say, in C++ i would have a method)
>     price.set_next_point(time, value);
>
> thanks!
>
> On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <[hidden email]>
> wrote:
>
>>
>>
>> > Wiadomość napisana przez Daniel Cegiełka <[hidden email]> w
>> dniu 06.09.2019, o godz. 16:10:
>> >
>>
>> >
>> > 2) preallocation
>> >
>> > preallocate_matrix <- function(n)
>> > {
>> >     x <- matrix()
>> >     length(x) <- 4 * n      # bid, ask, bid_size, ask_size
>> >     dim(x) <- c(n, 4)       # see: ?dim
>> >     return(x)
>> > }
>> >
>> > > x <- preallocate_matrix(5)
>> > > x
>> >      [,1] [,2] [,3] [,4]
>> > [1,]   NA   NA   NA   NA
>> > [2,]   NA   NA   NA   NA
>> > [3,]   NA   NA   NA   NA
>> > [4,]   NA   NA   NA   NA
>> > [5,]   NA   NA   NA   NA
>>
>> ?matrix
>>
>> Usage
>> matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
>>        dimnames = NULL)
>>
>> so we don't even need preallocate_matrix() function
>>
>> > x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
>> > x
>>                     [,1] [,2] [,3] [,4]
>> 2019-09-06 17:07:27   NA   NA   NA   NA
>> 2019-09-06 17:07:28   NA   NA   NA   NA
>> 2019-09-06 17:07:29   NA   NA   NA   NA
>> 2019-09-06 17:07:30   NA   NA   NA   NA
>> 2019-09-06 17:07:31   NA   NA   NA   NA
>>
>>
>>
>>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: how to grow XTS series in R dynamically ? And Quickly!

Vladimir Morozov-2
In reply to this post by Ilya Kipnis
Thanks, Ilya

On Sat, Sep 7, 2019, 3:47 AM Ilya Kipnis <[hidden email]> wrote:

> Vladimir,
>
> Question--do you have to do operations on the newly updated xts each time
> you append it? For instance, when I backtest strategies and I don't know
> how long the output would be, I start off by allocating an empty list,
> keep updating the list, and rbind it at the end.
>
> EG:
>
> results <- list()
> for(i in 1:n){
>   #output <- do something
>   results[[i]] <-output
> }
> results <- do.call(rbind, results)
>
> This might help, as this doesn't keep copying the larger list over and
> over again, which I remember doing before I learned about this method.
>
> Hope this helps.
>
> -Ilya
>
> On Fri, Sep 6, 2019 at 2:05 PM Vladimir Morozov <[hidden email]>
> wrote:
>
>> Hi Daniel
>> Thanks a lot.
>> Those are very helpful ideas.
>>
>> rbind_append --> it still has to allocate memory for the resulting
>> series... so if memory allocation was the main reason for slow
>> performance,
>> maybe rbind_append doesn't change much? what do you think?
>>
>> preallocating regular-interval time-series is a good idea.
>> however financial data are irregularly spaced (sometimes there may not be
>> any price updates for a few secs, even more).
>> even if we postulate that prices are allowed to change no more than once
>> per second, there's a lot of uses for the frequency of price updates, not
>> only the values of the prices (the simplest assumption is the poisson
>> arrival process for the updates, but there are many fancier, more powerful
>> models...)
>> so, pre-allocating a regularly spaced 1-sec interval xts series dumbs down
>> many things!
>>
>> i wish i could pre-allocate the vector for the values and maybe indices,
>> but then do the assignment of the sort:
>> (say, in C++ i would have a method)
>>     price.set_next_point(time, value);
>>
>> thanks!
>>
>> On Sat, Sep 7, 2019 at 12:08 AM Daniel Cegiełka <
>> [hidden email]>
>> wrote:
>>
>> >
>> >
>> > > Wiadomość napisana przez Daniel Cegiełka <[hidden email]>
>> w
>> > dniu 06.09.2019, o godz. 16:10:
>> > >
>> >
>> > >
>> > > 2) preallocation
>> > >
>> > > preallocate_matrix <- function(n)
>> > > {
>> > >     x <- matrix()
>> > >     length(x) <- 4 * n      # bid, ask, bid_size, ask_size
>> > >     dim(x) <- c(n, 4)       # see: ?dim
>> > >     return(x)
>> > > }
>> > >
>> > > > x <- preallocate_matrix(5)
>> > > > x
>> > >      [,1] [,2] [,3] [,4]
>> > > [1,]   NA   NA   NA   NA
>> > > [2,]   NA   NA   NA   NA
>> > > [3,]   NA   NA   NA   NA
>> > > [4,]   NA   NA   NA   NA
>> > > [5,]   NA   NA   NA   NA
>> >
>> > ?matrix
>> >
>> > Usage
>> > matrix(data = NA, nrow = 1, ncol = 1, byrow = FALSE,
>> >        dimnames = NULL)
>> >
>> > so we don't even need preallocate_matrix() function
>> >
>> > > x <- .xts(matrix(nrow = 5, ncol = 4), index = Sys.time() + 1:5)
>> > > x
>> >                     [,1] [,2] [,3] [,4]
>> > 2019-09-06 17:07:27   NA   NA   NA   NA
>> > 2019-09-06 17:07:28   NA   NA   NA   NA
>> > 2019-09-06 17:07:29   NA   NA   NA   NA
>> > 2019-09-06 17:07:30   NA   NA   NA   NA
>> > 2019-09-06 17:07:31   NA   NA   NA   NA
>> >
>> >
>> >
>> >
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R questions
>> should go.
>>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.