Vectorized expression to extrapolate matrix columns with columns of another matrix

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Vectorized expression to extrapolate matrix columns with columns of another matrix

Abiel X Reinhart
I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.

Thanks very much.

Abiel Reinhart
This communication is for informational purposes only. It is not
intended as an offer or solicitation for the purchase or sale of
any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not
warranted as to completeness or accuracy and are subject to change
without notice. Any comments or statements made herein do not
necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
and affiliates.

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.

Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to European legal entities.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Jeffrey J. Hallman-2
Abiel X Reinhart <[hidden email]> writes:

> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.

If you use tis objects and all of the columns of mat1 become NA on the same row, i.e.,
mat 1 looks like this:

20100101  1  2  3
20100102  4  5  6
20100103 NA NA NA
20100104 NA NA NA

then something like this should work:

mat1 <- naWindow(mat1)  ## drops the ending NA rows
rate <- growth.rate(window(mat2, start = end(mat1)))
growth.rate(mat1) <- rate

--
Jeff

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Gabor Grothendieck
In reply to this post by Abiel X Reinhart
Try this using the zoo package.  See ?na.approx for more and note that
this functionality requires zoo 1.6-3 or later.

.> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
> na.approx(m[, 1], x = m[, 2])
1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
> na.approx(m[, 1])
1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
         1          2          3          4          5          6          7


On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
<[hidden email]> wrote:

> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>
> Thanks very much.
>
> Abiel Reinhart
> This communication is for informational purposes only. It is not
> intended as an offer or solicitation for the purchase or sale of
> any financial instrument or as an official confirmation of any
> transaction. All market prices, data and other information are not
> warranted as to completeness or accuracy and are subject to change
> without notice. Any comments or statements made herein do not
> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
> and affiliates.
>
> This transmission may contain information that is privileged,
> confidential, legally privileged, and/or exempt from disclosure
> under applicable law. If you are not the intended recipient, you
> are hereby notified that any disclosure, copying, distribution, or
> use of the information contained herein (including any reliance
> thereon) is STRICTLY PROHIBITED. Although this transmission and any
> attachments are believed to be free of any virus or other defect
> that might affect any computer system into which it is received and
> opened, it is the responsibility of the recipient to ensure that it
> is virus free and no responsibility is accepted by JPMorgan Chase &
> Co., its subsidiaries and affiliates, as applicable, for any loss
> or damage arising in any way from its use. If you received this
> transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard
> copy format. Thank you.
>
> Please refer to http://www.jpmorgan.com/pages/disclosures for
> disclosures relating to European legal entities.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Abiel X Reinhart
Gabor,

This comes close to solving my problem, but I am still left with the problem of how I can extrapolate, not just interpolate. In our example, if I define m as,

m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))

instead of

m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))

then I will only get five values back, when I really want m[,1] to be fully extrapolated so that there are seven values. Is there a workaround?

Abiel Reinhart

-----Original Message-----
From: Gabor Grothendieck [mailto:[hidden email]]
Sent: Wednesday, May 12, 2010 8:37 AM
To: Abiel X Reinhart
Cc: [hidden email]
Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix

Try this using the zoo package.  See ?na.approx for more and note that
this functionality requires zoo 1.6-3 or later.

.> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
> na.approx(m[, 1], x = m[, 2])
1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
> na.approx(m[, 1])
1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
         1          2          3          4          5          6          7


On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
<[hidden email]> wrote:

> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>
> Thanks very much.
>
> Abiel Reinhart
> This communication is for informational purposes only. It is not
> intended as an offer or solicitation for the purchase or sale of
> any financial instrument or as an official confirmation of any
> transaction. All market prices, data and other information are not
> warranted as to completeness or accuracy and are subject to change
> without notice. Any comments or statements made herein do not
> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
> and affiliates.
>
> This transmission may contain information that is privileged,
> confidential, legally privileged, and/or exempt from disclosure
> under applicable law. If you are not the intended recipient, you
> are hereby notified that any disclosure, copying, distribution, or
> use of the information contained herein (including any reliance
> thereon) is STRICTLY PROHIBITED. Although this transmission and any
> attachments are believed to be free of any virus or other defect
> that might affect any computer system into which it is received and
> opened, it is the responsibility of the recipient to ensure that it
> is virus free and no responsibility is accepted by JPMorgan Chase &
> Co., its subsidiaries and affiliates, as applicable, for any loss
> or damage arising in any way from its use. If you received this
> transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard
> copy format. Thank you.
>
> Please refer to http://www.jpmorgan.com/pages/disclosures for
> disclosures relating to European legal entities.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Gabor Grothendieck
Use rule = 2 as in the extrapolation examples in the na.approx help file.

On Wed, May 12, 2010 at 10:10 AM, Abiel X Reinhart
<[hidden email]> wrote:

> Gabor,
>
> This comes close to solving my problem, but I am still left with the problem of how I can extrapolate, not just interpolate. In our example, if I define m as,
>
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>
> instead of
>
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>
> then I will only get five values back, when I really want m[,1] to be fully extrapolated so that there are seven values. Is there a workaround?
>
> Abiel Reinhart
>
> -----Original Message-----
> From: Gabor Grothendieck [mailto:[hidden email]]
> Sent: Wednesday, May 12, 2010 8:37 AM
> To: Abiel X Reinhart
> Cc: [hidden email]
> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>
> Try this using the zoo package.  See ?na.approx for more and note that
> this functionality requires zoo 1.6-3 or later.
>
> .> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>> na.approx(m[, 1], x = m[, 2])
> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
>> na.approx(m[, 1])
> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>         1          2          3          4          5          6          7
>
>
> On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
> <[hidden email]> wrote:
>> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>>
>> Thanks very much.
>>
>> Abiel Reinhart
>> This communication is for informational purposes only. It is not
>> intended as an offer or solicitation for the purchase or sale of
>> any financial instrument or as an official confirmation of any
>> transaction. All market prices, data and other information are not
>> warranted as to completeness or accuracy and are subject to change
>> without notice. Any comments or statements made herein do not
>> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
>> and affiliates.
>>
>> This transmission may contain information that is privileged,
>> confidential, legally privileged, and/or exempt from disclosure
>> under applicable law. If you are not the intended recipient, you
>> are hereby notified that any disclosure, copying, distribution, or
>> use of the information contained herein (including any reliance
>> thereon) is STRICTLY PROHIBITED. Although this transmission and any
>> attachments are believed to be free of any virus or other defect
>> that might affect any computer system into which it is received and
>> opened, it is the responsibility of the recipient to ensure that it
>> is virus free and no responsibility is accepted by JPMorgan Chase &
>> Co., its subsidiaries and affiliates, as applicable, for any loss
>> or damage arising in any way from its use. If you received this
>> transmission in error, please immediately contact the sender and
>> destroy the material in its entirety, whether in electronic or hard
>> copy format. Thank you.
>>
>> Please refer to http://www.jpmorgan.com/pages/disclosures for
>> disclosures relating to European legal entities.
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Abiel X Reinhart
Gabor,

Maybe I am doing this wrong, but rule=2 does not look like it is growing the series out, but rather just carrying the last value forward. It looks like na.approx() followed by na.locf(). For instance:

m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
na.approx(m[, 1], x = m[, 2], rule=2)

1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
       1.0        2.0        2.7        3.7        5.0        5.0        5.0

Abiel Reinhart

-----Original Message-----
From: Gabor Grothendieck [mailto:[hidden email]]
Sent: Wednesday, May 12, 2010 10:40 AM
To: Abiel X Reinhart
Cc: [hidden email]
Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix

Use rule = 2 as in the extrapolation examples in the na.approx help file.

On Wed, May 12, 2010 at 10:10 AM, Abiel X Reinhart
<[hidden email]> wrote:

> Gabor,
>
> This comes close to solving my problem, but I am still left with the problem of how I can extrapolate, not just interpolate. In our example, if I define m as,
>
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>
> instead of
>
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>
> then I will only get five values back, when I really want m[,1] to be fully extrapolated so that there are seven values. Is there a workaround?
>
> Abiel Reinhart
>
> -----Original Message-----
> From: Gabor Grothendieck [mailto:[hidden email]]
> Sent: Wednesday, May 12, 2010 8:37 AM
> To: Abiel X Reinhart
> Cc: [hidden email]
> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>
> Try this using the zoo package.  See ?na.approx for more and note that
> this functionality requires zoo 1.6-3 or later.
>
> .> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>> na.approx(m[, 1], x = m[, 2])
> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
>> na.approx(m[, 1])
> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>         1          2          3          4          5          6          7
>
>
> On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
> <[hidden email]> wrote:
>> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>>
>> Thanks very much.
>>
>> Abiel Reinhart
>> This communication is for informational purposes only. It is not
>> intended as an offer or solicitation for the purchase or sale of
>> any financial instrument or as an official confirmation of any
>> transaction. All market prices, data and other information are not
>> warranted as to completeness or accuracy and are subject to change
>> without notice. Any comments or statements made herein do not
>> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
>> and affiliates.
>>
>> This transmission may contain information that is privileged,
>> confidential, legally privileged, and/or exempt from disclosure
>> under applicable law. If you are not the intended recipient, you
>> are hereby notified that any disclosure, copying, distribution, or
>> use of the information contained herein (including any reliance
>> thereon) is STRICTLY PROHIBITED. Although this transmission and any
>> attachments are believed to be free of any virus or other defect
>> that might affect any computer system into which it is received and
>> opened, it is the responsibility of the recipient to ensure that it
>> is virus free and no responsibility is accepted by JPMorgan Chase &
>> Co., its subsidiaries and affiliates, as applicable, for any loss
>> or damage arising in any way from its use. If you received this
>> transmission in error, please immediately contact the sender and
>> destroy the material in its entirety, whether in electronic or hard
>> copy format. Thank you.
>>
>> Please refer to http://www.jpmorgan.com/pages/disclosures for
>> disclosures relating to European legal entities.
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Gabor Grothendieck
Yes, that is what it does.  Note that na.approx interpolates and does
not work precisely as you discussed but its easy, does use m[,2] and
may be good enough.   If you really do want something precisely as you
discussed try this.  It NAs out the rows of m for which column 1 is NA
and then uses na.locf to move the prior non-NA into it.  Then we apply
the formula:

> library(zoo)
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>
> # mm will hold result; m.locf
> m.locf <- mm <- m
> ix <- is.na(mm[,1])
> m.locf[ix,] <- NA
> m.locf <- na.locf(m.locf)
> mm[ix, 1] <- m.locf[ix, 1] * mm[ix,2] / m.locf[ix,2]
> mm

1970-01-02 1.0  1
1970-01-03 2.0  4
1970-01-04 4.5  9
1970-01-05 8.0 16
1970-01-06 5.0 25
1970-01-07 7.2 36
1970-01-08 9.8 49

1970-01-02 1.0  1
1970-01-03 2.0  4
1970-01-04 4.5  9
1970-01-05 8.0 16
1970-01-06 5.0 25
1970-01-07 7.2 36
1970-01-08 9.8 49





Yes, that is what it does. Please read the help file for na.approx and
approx.  If you want something different you will have to special case
the end values.

On Wed, May 12, 2010 at 11:11 AM, Abiel X Reinhart
<[hidden email]> wrote:

> Gabor,
>
> Maybe I am doing this wrong, but rule=2 does not look like it is growing the series out, but rather just carrying the last value forward. It looks like na.approx() followed by na.locf(). For instance:
>
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
> na.approx(m[, 1], x = m[, 2], rule=2)
>
> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>       1.0        2.0        2.7        3.7        5.0        5.0        5.0
>
> Abiel Reinhart
>
> -----Original Message-----
> From: Gabor Grothendieck [mailto:[hidden email]]
> Sent: Wednesday, May 12, 2010 10:40 AM
> To: Abiel X Reinhart
> Cc: [hidden email]
> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>
> Use rule = 2 as in the extrapolation examples in the na.approx help file.
>
> On Wed, May 12, 2010 at 10:10 AM, Abiel X Reinhart
> <[hidden email]> wrote:
>> Gabor,
>>
>> This comes close to solving my problem, but I am still left with the problem of how I can extrapolate, not just interpolate. In our example, if I define m as,
>>
>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>>
>> instead of
>>
>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>>
>> then I will only get five values back, when I really want m[,1] to be fully extrapolated so that there are seven values. Is there a workaround?
>>
>> Abiel Reinhart
>>
>> -----Original Message-----
>> From: Gabor Grothendieck [mailto:[hidden email]]
>> Sent: Wednesday, May 12, 2010 8:37 AM
>> To: Abiel X Reinhart
>> Cc: [hidden email]
>> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>>
>> Try this using the zoo package.  See ?na.approx for more and note that
>> this functionality requires zoo 1.6-3 or later.
>>
>> .> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>>> na.approx(m[, 1], x = m[, 2])
>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
>>> na.approx(m[, 1])
>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>         1          2          3          4          5          6          7
>>
>>
>> On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
>> <[hidden email]> wrote:
>>> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>>>
>>> Thanks very much.
>>>
>>> Abiel Reinhart
>>> This communication is for informational purposes only. It is not
>>> intended as an offer or solicitation for the purchase or sale of
>>> any financial instrument or as an official confirmation of any
>>> transaction. All market prices, data and other information are not
>>> warranted as to completeness or accuracy and are subject to change
>>> without notice. Any comments or statements made herein do not
>>> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
>>> and affiliates.
>>>
>>> This transmission may contain information that is privileged,
>>> confidential, legally privileged, and/or exempt from disclosure
>>> under applicable law. If you are not the intended recipient, you
>>> are hereby notified that any disclosure, copying, distribution, or
>>> use of the information contained herein (including any reliance
>>> thereon) is STRICTLY PROHIBITED. Although this transmission and any
>>> attachments are believed to be free of any virus or other defect
>>> that might affect any computer system into which it is received and
>>> opened, it is the responsibility of the recipient to ensure that it
>>> is virus free and no responsibility is accepted by JPMorgan Chase &
>>> Co., its subsidiaries and affiliates, as applicable, for any loss
>>> or damage arising in any way from its use. If you received this
>>> transmission in error, please immediately contact the sender and
>>> destroy the material in its entirety, whether in electronic or hard
>>> copy format. Thank you.
>>>
>>> Please refer to http://www.jpmorgan.com/pages/disclosures for
>>> disclosures relating to European legal entities.
>>>
>>> ______________________________________________
>>> [hidden email] mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Abiel X Reinhart
Thanks Gabor, this looks like it serves my needs. I've extended the code to work with an example where we have two multicolumn zoo objects, one with the original data and another that has the growth rates.

# mat1 = zoo object to extend
# mat2 = zoo object whose growth rate is used to extend mat1
mergeGrowth <- function(mat1, mat2)
{
        ix <- is.na(mat1)
        mat1.locf <- na.locf(mat1, na.rm=F)
        mat2.locf <- mat2
        mat2.locf[ix] <- NA
        mat2.locf <- na.locf(mat2.locf, na.rm=F)
        coredata(mat1)[ix] <- coredata(mat1.locf * mat2 / mat2.locf)[ix]
        mat1
}


Abiel Reinhart

-----Original Message-----
From: Gabor Grothendieck [mailto:[hidden email]]
Sent: Wednesday, May 12, 2010 12:00 PM
To: Abiel X Reinhart
Cc: [hidden email]
Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix

Yes, that is what it does.  Note that na.approx interpolates and does
not work precisely as you discussed but its easy, does use m[,2] and
may be good enough.   If you really do want something precisely as you
discussed try this.  It NAs out the rows of m for which column 1 is NA
and then uses na.locf to move the prior non-NA into it.  Then we apply
the formula:

> library(zoo)
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>
> # mm will hold result; m.locf
> m.locf <- mm <- m
> ix <- is.na(mm[,1])
> m.locf[ix,] <- NA
> m.locf <- na.locf(m.locf)
> mm[ix, 1] <- m.locf[ix, 1] * mm[ix,2] / m.locf[ix,2]
> mm

1970-01-02 1.0  1
1970-01-03 2.0  4
1970-01-04 4.5  9
1970-01-05 8.0 16
1970-01-06 5.0 25
1970-01-07 7.2 36
1970-01-08 9.8 49

1970-01-02 1.0  1
1970-01-03 2.0  4
1970-01-04 4.5  9
1970-01-05 8.0 16
1970-01-06 5.0 25
1970-01-07 7.2 36
1970-01-08 9.8 49





Yes, that is what it does. Please read the help file for na.approx and
approx.  If you want something different you will have to special case
the end values.

On Wed, May 12, 2010 at 11:11 AM, Abiel X Reinhart
<[hidden email]> wrote:

> Gabor,
>
> Maybe I am doing this wrong, but rule=2 does not look like it is growing the series out, but rather just carrying the last value forward. It looks like na.approx() followed by na.locf(). For instance:
>
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
> na.approx(m[, 1], x = m[, 2], rule=2)
>
> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>       1.0        2.0        2.7        3.7        5.0        5.0        5.0
>
> Abiel Reinhart
>
> -----Original Message-----
> From: Gabor Grothendieck [mailto:[hidden email]]
> Sent: Wednesday, May 12, 2010 10:40 AM
> To: Abiel X Reinhart
> Cc: [hidden email]
> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>
> Use rule = 2 as in the extrapolation examples in the na.approx help file.
>
> On Wed, May 12, 2010 at 10:10 AM, Abiel X Reinhart
> <[hidden email]> wrote:
>> Gabor,
>>
>> This comes close to solving my problem, but I am still left with the problem of how I can extrapolate, not just interpolate. In our example, if I define m as,
>>
>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>>
>> instead of
>>
>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>>
>> then I will only get five values back, when I really want m[,1] to be fully extrapolated so that there are seven values. Is there a workaround?
>>
>> Abiel Reinhart
>>
>> -----Original Message-----
>> From: Gabor Grothendieck [mailto:[hidden email]]
>> Sent: Wednesday, May 12, 2010 8:37 AM
>> To: Abiel X Reinhart
>> Cc: [hidden email]
>> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>>
>> Try this using the zoo package.  See ?na.approx for more and note that
>> this functionality requires zoo 1.6-3 or later.
>>
>> .> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>>> na.approx(m[, 1], x = m[, 2])
>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
>>> na.approx(m[, 1])
>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>         1          2          3          4          5          6          7
>>
>>
>> On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
>> <[hidden email]> wrote:
>>> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>>>
>>> Thanks very much.
>>>
>>> Abiel Reinhart
>>> This communication is for informational purposes only. It is not
>>> intended as an offer or solicitation for the purchase or sale of
>>> any financial instrument or as an official confirmation of any
>>> transaction. All market prices, data and other information are not
>>> warranted as to completeness or accuracy and are subject to change
>>> without notice. Any comments or statements made herein do not
>>> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
>>> and affiliates.
>>>
>>> This transmission may contain information that is privileged,
>>> confidential, legally privileged, and/or exempt from disclosure
>>> under applicable law. If you are not the intended recipient, you
>>> are hereby notified that any disclosure, copying, distribution, or
>>> use of the information contained herein (including any reliance
>>> thereon) is STRICTLY PROHIBITED. Although this transmission and any
>>> attachments are believed to be free of any virus or other defect
>>> that might affect any computer system into which it is received and
>>> opened, it is the responsibility of the recipient to ensure that it
>>> is virus free and no responsibility is accepted by JPMorgan Chase &
>>> Co., its subsidiaries and affiliates, as applicable, for any loss
>>> or damage arising in any way from its use. If you received this
>>> transmission in error, please immediately contact the sender and
>>> destroy the material in its entirety, whether in electronic or hard
>>> copy format. Thank you.
>>>
>>> Please refer to http://www.jpmorgan.com/pages/disclosures for
>>> disclosures relating to European legal entities.
>>>
>>> ______________________________________________
>>> [hidden email] mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Vectorized expression to extrapolate matrix columns with columns of another matrix

Gabor Grothendieck
I have managed to reduce it to two statements using the trick that the
elements of x-x are NA or 0 according to whether the corresponding
element of x is NA or not.  You could probably extend this to your
situation too.

> library(zoo)
> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>
> r <- replace(m[,1], TRUE,
+                  na.locf(m[,1]) * m[,2] / na.locf(m[,2] + (m[,1]-m[,1])))
> cbind(V1 = r, V2 = m[,2])
            V1 V2
1970-01-02 1.0  1
1970-01-03 2.0  4
1970-01-04 4.5  9
1970-01-05 8.0 16
1970-01-06 5.0 25
1970-01-07 7.2 36
1970-01-08 9.8 49


On Wed, May 12, 2010 at 4:58 PM, Abiel X Reinhart
<[hidden email]> wrote:

> Thanks Gabor, this looks like it serves my needs. I've extended the code to work with an example where we have two multicolumn zoo objects, one with the original data and another that has the growth rates.
>
> # mat1 = zoo object to extend
> # mat2 = zoo object whose growth rate is used to extend mat1
> mergeGrowth <- function(mat1, mat2)
> {
>        ix <- is.na(mat1)
>        mat1.locf <- na.locf(mat1, na.rm=F)
>        mat2.locf <- mat2
>        mat2.locf[ix] <- NA
>        mat2.locf <- na.locf(mat2.locf, na.rm=F)
>        coredata(mat1)[ix] <- coredata(mat1.locf * mat2 / mat2.locf)[ix]
>        mat1
> }
>
>
> Abiel Reinhart
>
> -----Original Message-----
> From: Gabor Grothendieck [mailto:[hidden email]]
> Sent: Wednesday, May 12, 2010 12:00 PM
> To: Abiel X Reinhart
> Cc: [hidden email]
> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>
> Yes, that is what it does.  Note that na.approx interpolates and does
> not work precisely as you discussed but its easy, does use m[,2] and
> may be good enough.   If you really do want something precisely as you
> discussed try this.  It NAs out the rows of m for which column 1 is NA
> and then uses na.locf to move the prior non-NA into it.  Then we apply
> the formula:
>
>> library(zoo)
>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>>
>> # mm will hold result; m.locf
>> m.locf <- mm <- m
>> ix <- is.na(mm[,1])
>> m.locf[ix,] <- NA
>> m.locf <- na.locf(m.locf)
>> mm[ix, 1] <- m.locf[ix, 1] * mm[ix,2] / m.locf[ix,2]
>> mm
>
> 1970-01-02 1.0  1
> 1970-01-03 2.0  4
> 1970-01-04 4.5  9
> 1970-01-05 8.0 16
> 1970-01-06 5.0 25
> 1970-01-07 7.2 36
> 1970-01-08 9.8 49
>
> 1970-01-02 1.0  1
> 1970-01-03 2.0  4
> 1970-01-04 4.5  9
> 1970-01-05 8.0 16
> 1970-01-06 5.0 25
> 1970-01-07 7.2 36
> 1970-01-08 9.8 49
>
>
>
>
>
> Yes, that is what it does. Please read the help file for na.approx and
> approx.  If you want something different you will have to special case
> the end values.
>
> On Wed, May 12, 2010 at 11:11 AM, Abiel X Reinhart
> <[hidden email]> wrote:
>> Gabor,
>>
>> Maybe I am doing this wrong, but rule=2 does not look like it is growing the series out, but rather just carrying the last value forward. It looks like na.approx() followed by na.locf(). For instance:
>>
>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>> na.approx(m[, 1], x = m[, 2], rule=2)
>>
>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>       1.0        2.0        2.7        3.7        5.0        5.0        5.0
>>
>> Abiel Reinhart
>>
>> -----Original Message-----
>> From: Gabor Grothendieck [mailto:[hidden email]]
>> Sent: Wednesday, May 12, 2010 10:40 AM
>> To: Abiel X Reinhart
>> Cc: [hidden email]
>> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>>
>> Use rule = 2 as in the extrapolation examples in the na.approx help file.
>>
>> On Wed, May 12, 2010 at 10:10 AM, Abiel X Reinhart
>> <[hidden email]> wrote:
>>> Gabor,
>>>
>>> This comes close to solving my problem, but I am still left with the problem of how I can extrapolate, not just interpolate. In our example, if I define m as,
>>>
>>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, NA), seq(7)^2), as.Date(1:7))
>>>
>>> instead of
>>>
>>> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>>>
>>> then I will only get five values back, when I really want m[,1] to be fully extrapolated so that there are seven values. Is there a workaround?
>>>
>>> Abiel Reinhart
>>>
>>> -----Original Message-----
>>> From: Gabor Grothendieck [mailto:[hidden email]]
>>> Sent: Wednesday, May 12, 2010 8:37 AM
>>> To: Abiel X Reinhart
>>> Cc: [hidden email]
>>> Subject: Re: [R] Vectorized expression to extrapolate matrix columns with columns of another matrix
>>>
>>> Try this using the zoo package.  See ?na.approx for more and note that
>>> this functionality requires zoo 1.6-3 or later.
>>>
>>> .> m <- zoo(cbind(c(1, 2, NA, NA, 5, NA, 7), seq(7)^2), as.Date(1:7))
>>>> na.approx(m[, 1], x = m[, 2])
>>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>>  1.000000   2.000000   2.714286   3.714286   5.000000   5.916667   7.000000
>>>> na.approx(m[, 1])
>>> 1970-01-02 1970-01-03 1970-01-04 1970-01-05 1970-01-06 1970-01-07 1970-01-08
>>>         1          2          3          4          5          6          7
>>>
>>>
>>> On Tue, May 11, 2010 at 4:48 PM, Abiel X Reinhart
>>> <[hidden email]> wrote:
>>>> I have two identically sized matrices of data that represent time series (I am storing the data in zoo objects, but the idea should apply to any matrix of data). The time series in the second matrix extend further than in the first matrix, and I would like to use the data in matrix 2 to extrapolate the data in matrix 1. In other words, if mat1[i,j] == NA, then mat1[i,j] <- mat1[i-1, j]*mat2[i,j]/mat2[i-1,j]. Of course, before we can calculate mat1[i,j] we may need to calculate mat1[i-1,j], and that in turn may require the computation of mat1[i-2,j], etc. This could all clearly be done with loops, but I am wondering if anyone can think of a vectorized expression or other efficient method that would work.
>>>>
>>>> Thanks very much.
>>>>
>>>> Abiel Reinhart
>>>> This communication is for informational purposes only. It is not
>>>> intended as an offer or solicitation for the purchase or sale of
>>>> any financial instrument or as an official confirmation of any
>>>> transaction. All market prices, data and other information are not
>>>> warranted as to completeness or accuracy and are subject to change
>>>> without notice. Any comments or statements made herein do not
>>>> necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
>>>> and affiliates.
>>>>
>>>> This transmission may contain information that is privileged,
>>>> confidential, legally privileged, and/or exempt from disclosure
>>>> under applicable law. If you are not the intended recipient, you
>>>> are hereby notified that any disclosure, copying, distribution, or
>>>> use of the information contained herein (including any reliance
>>>> thereon) is STRICTLY PROHIBITED. Although this transmission and any
>>>> attachments are believed to be free of any virus or other defect
>>>> that might affect any computer system into which it is received and
>>>> opened, it is the responsibility of the recipient to ensure that it
>>>> is virus free and no responsibility is accepted by JPMorgan Chase &
>>>> Co., its subsidiaries and affiliates, as applicable, for any loss
>>>> or damage arising in any way from its use. If you received this
>>>> transmission in error, please immediately contact the sender and
>>>> destroy the material in its entirety, whether in electronic or hard
>>>> copy format. Thank you.
>>>>
>>>> Please refer to http://www.jpmorgan.com/pages/disclosures for
>>>> disclosures relating to European legal entities.
>>>>
>>>> ______________________________________________
>>>> [hidden email] mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>
>>>
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.