

i am a beginner regarding R but i am trying to do a simple thing, but it is
taking too much time and i am asking if there is any way to achieve what i
need, i have a time series data set with 730 data points, i detected 7, 354
and 365 seasonality periods. i am trying to use Fourier terms for
seasonality and for loop to get the K value for each while minimizing AICc,
my code is
AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
for (j in 1:177) {
for (k in 182) { #i,j and k values are choosen
with regad that K cannot exceed seasonality period/2
z1 < fourier(ts(demand,frequency = 7), K=i)
z2 < fourier(ts(demand,frequency=354), K=j)
z3 < fourier(ts(demand,frequency = 365),K=k)
fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
seasonal = FALSE)
fit$aicc
AICc[,1]<i
AICc[,2]<j
AICc[,3]<k
AICc[,4]<fit$aicc
}
}
}
AICc
i have created a data table to store AICc values from all possible i,j,k
combinations so that i can find later the minimum AICc value. the problem
now is that it is taking forever to do so not only to iterate all
combinations but also due to the large K values.
, is there any possible solution for this? thank you in advance
[[alternative HTML version deleted]]
______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.


This is not an answer to your speed problem but are your assignments to
AICc[,1] and so on doing what you hope they are doing?
Michael
On 06/02/2019 12:03, salah maadawy wrote:
> i am a beginner regarding R but i am trying to do a simple thing, but it is
> taking too much time and i am asking if there is any way to achieve what i
> need, i have a time series data set with 730 data points, i detected 7, 354
> and 365 seasonality periods. i am trying to use Fourier terms for
> seasonality and for loop to get the K value for each while minimizing AICc,
> my code is
>
> AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
> for (j in 1:177) {
> for (k in 182) { #i,j and k values are choosen
> with regad that K cannot exceed seasonality period/2
> z1 < fourier(ts(demand,frequency = 7), K=i)
> z2 < fourier(ts(demand,frequency=354), K=j)
> z3 < fourier(ts(demand,frequency = 365),K=k)
> fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> seasonal = FALSE)
> fit$aicc
> AICc[,1]<i
> AICc[,2]<j
> AICc[,3]<k
> AICc[,4]<fit$aicc
> }
>
> }
> }
> AICc
>
> i have created a data table to store AICc values from all possible i,j,k
> combinations so that i can find later the minimum AICc value. the problem
> now is that it is taking forever to do so not only to iterate all
> combinations but also due to the large K values.
>
> , is there any possible solution for this? thank you in advance
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list  To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/rhelp> PLEASE do read the posting guide http://www.Rproject.org/postingguide.html> and provide commented, minimal, selfcontained, reproducible code.
>

Michael
http://www.dewey.myzen.co.uk/home.html______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.


Hi Micheal, Maybe there is a simple way but i wanted to get the lowest aicc
ana i could not find a way to do so, that's why i created the table to
store all possible outcomes and then i can easily get the minimum value and
the values of (i,j and k) used for that minimum value. The first column in
the table is AICc[,1] to store i and second column for j and so on. Maybe i
am mistaken and this won't give me what i want, the code been running for 5
hours now. So i am waiting
On Wed, Feb 6, 2019 at 4:59 PM Michael Dewey < [hidden email]>
wrote:
> This is not an answer to your speed problem but are your assignments to
> AICc[,1] and so on doing what you hope they are doing?
>
> Michael
>
> On 06/02/2019 12:03, salah maadawy wrote:
> > i am a beginner regarding R but i am trying to do a simple thing, but it
> is
> > taking too much time and i am asking if there is any way to achieve what
> i
> > need, i have a time series data set with 730 data points, i detected 7,
> 354
> > and 365 seasonality periods. i am trying to use Fourier terms for
> > seasonality and for loop to get the K value for each while minimizing
> AICc,
> > my code is
> >
> > AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
> > for (j in 1:177) {
> > for (k in 182) { #i,j and k values are choosen
> > with regad that K cannot exceed seasonality period/2
> > z1 < fourier(ts(demand,frequency = 7), K=i)
> > z2 < fourier(ts(demand,frequency=354), K=j)
> > z3 < fourier(ts(demand,frequency = 365),K=k)
> > fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> > seasonal = FALSE)
> > fit$aicc
> > AICc[,1]<i
> > AICc[,2]<j
> > AICc[,3]<k
> > AICc[,4]<fit$aicc
> > }
> >
> > }
> > }
> > AICc
> >
> > i have created a data table to store AICc values from all possible i,j,k
> > combinations so that i can find later the minimum AICc value. the problem
> > now is that it is taking forever to do so not only to iterate all
> > combinations but also due to the large K values.
> >
> > , is there any possible solution for this? thank you in advance
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list  To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/rhelp> > PLEASE do read the posting guide
> http://www.Rproject.org/postingguide.html> > and provide commented, minimal, selfcontained, reproducible code.
> >
>
> 
> Michael
> http://www.dewey.myzen.co.uk/home.html>
[[alternative HTML version deleted]]
______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.


This seems like an odd analysis to me, but I don't have time to look
closer at it. It certainly doesn't look simple to me... I hope you have
some good theoretical guidance in tackling this (which this mailing list
is not).
One obvious thing is that z1 only depends on i, so you should only have to
compute z1 3 times... but you are computing it 177*182*3 times. Make three
lists of fourier results using three separate preparation loops before
diving into your triple loop. Then just retrieve the relevant z1, z2 and
z3 from these lists when you run the auto.arima calculation.
The easy way to make lists is to use lapply rather than a for loop:
demand7 < ts( demand, frequency = 7 ) # don't repeat the ts conversion
z1List < lapply( 1:3, function(i) { fourier( demand7, K = i ) } )
then you can retrieve whichever z1 you want with
z1 < z1List[[ i ]]
rather than recalculating it.
Also, data.table is implemented by a contributed package. It is a special
type of data frame. It is not at all clear that you need its special
properties here, but most people who create a matrix and then immediately
converts the matrix to a (type of) data frame aren't using the data frame
properly anyway. Here is one way to assemble your results:
AICc < expand.grid( k = 1:181, j = 1:177, i = 1:3 )
AICc$fit < NA
idx < 1
for ( i in 1:3 ) {
for ( j in 1:177 ) {
for ( k in 1:181 ) {
# compute fit
AICc$fit[ idx ] < fit$aicc
idx < idx + 1
}
}
}
On Wed, 6 Feb 2019, salah maadawy wrote:
> i am a beginner regarding R but i am trying to do a simple thing, but it is
> taking too much time and i am asking if there is any way to achieve what i
> need, i have a time series data set with 730 data points, i detected 7, 354
> and 365 seasonality periods. i am trying to use Fourier terms for
> seasonality and for loop to get the K value for each while minimizing AICc,
> my code is
>
> AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
> for (j in 1:177) {
> for (k in 182) { #i,j and k values are choosen
> with regad that K cannot exceed seasonality period/2
> z1 < fourier(ts(demand,frequency = 7), K=i)
> z2 < fourier(ts(demand,frequency=354), K=j)
> z3 < fourier(ts(demand,frequency = 365),K=k)
> fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> seasonal = FALSE)
> fit$aicc
> AICc[,1]<i
> AICc[,2]<j
> AICc[,3]<k
> AICc[,4]<fit$aicc
> }
>
> }
> }
> AICc
>
> i have created a data table to store AICc values from all possible i,j,k
> combinations so that i can find later the minimum AICc value. the problem
> now is that it is taking forever to do so not only to iterate all
> combinations but also due to the large K values.
>
> , is there any possible solution for this? thank you in advance
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list  To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/rhelp> PLEASE do read the posting guide http://www.Rproject.org/postingguide.html> and provide commented, minimal, selfcontained, reproducible code.
>

Jeff Newmiller The ..... ..... Go Live...
DCN:< [hidden email]> Basics: ##.#. ##.#. Live Go...
Live: OO#.. Dead: OO#.. Playing
Research Engineer (Solar/Batteries O.O#. #.O#. with
/Software/Embedded Controllers) .OO#. .OO#. rocks...1k
______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.


Well I do not know about data.table but in standard R if you go
AICc[,1] < 3
it fills the whole column with 3 so you will end up with a table with
the last value of AICc stored in every row which is almost certainly not
what you want.
Michael
On 06/02/2019 14:15, salah maadawy wrote:
> Hi Micheal, Maybe there is a simple way but i wanted to get the lowest
> aicc ana i could not find a way to do so, that's why i created the
> table to store all possible outcomes and then i can easily get the
> minimum value and the values of (i,j and k) used for that minimum value.
> The first column in the table is AICc[,1] to store i and second column
> for j and so on. Maybe i am mistaken and this won't give me what i want,
> the code been running for 5 hours now. So i am waiting
>
> On Wed, Feb 6, 2019 at 4:59 PM Michael Dewey < [hidden email]
> <mailto: [hidden email]>> wrote:
>
> This is not an answer to your speed problem but are your assignments to
> AICc[,1] and so on doing what you hope they are doing?
>
> Michael
>
> On 06/02/2019 12:03, salah maadawy wrote:
> > i am a beginner regarding R but i am trying to do a simple thing,
> but it is
> > taking too much time and i am asking if there is any way to
> achieve what i
> > need, i have a time series data set with 730 data points, i
> detected 7, 354
> > and 365 seasonality periods. i am trying to use Fourier terms for
> > seasonality and for loop to get the K value for each while
> minimizing AICc,
> > my code is
> >
> > AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in
> 1:3) {
> > for (j in 1:177) {
> > for (k in 182) { #i,j and k values are
> choosen
> > with regad that K cannot exceed seasonality period/2
> > z1 < fourier(ts(demand,frequency = 7), K=i)
> > z2 < fourier(ts(demand,frequency=354), K=j)
> > z3 < fourier(ts(demand,frequency = 365),K=k)
> > fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> > seasonal = FALSE)
> > fit$aicc
> > AICc[,1]<i
> > AICc[,2]<j
> > AICc[,3]<k
> > AICc[,4]<fit$aicc
> > }
> >
> > }
> > }
> > AICc
> >
> > i have created a data table to store AICc values from all
> possible i,j,k
> > combinations so that i can find later the minimum AICc value. the
> problem
> > now is that it is taking forever to do so not only to iterate all
> > combinations but also due to the large K values.
> >
> > , is there any possible solution for this? thank you in advance
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] <mailto: [hidden email]> mailing list
>  To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/rhelp> > PLEASE do read the posting guide
> http://www.Rproject.org/postingguide.html> > and provide commented, minimal, selfcontained, reproducible code.
> >
>
> 
> Michael
> http://www.dewey.myzen.co.uk/home.html>

Michael
http://www.dewey.myzen.co.uk/home.html______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.


With 96k model fits it's going to be slow, so you might want to think
first about whether you need to do them all. Beyond that, I think this
is more in the R style, so might be quicker (I don't know how much the
loops are slowing you down), and even if not it should be easier to
adapt.
The other thing to think about is parallelising the code  the
parallel package should help.
FitModel < function(K, data) {
z1 < fourier(ts(data,frequency = 7), K=K["i"])
z2 < fourier(ts(data,frequency=354), K=K["j"])
z3 < fourier(ts(data,,frequency = 365),K=K["k"])
fit < auto.arima(data,, xreg =cbind(z1,z2,z3), seasonal = FALSE)
fit$aicc
}
# smaller MaxOrders used so if you run it like this, it won't take 5 hours
MaxOrders < expand.grid(i = 1:3, j=1=7, k=1:8)
AICc < apply(MaxOrders, FitModel, data=demand)
Bob
AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
> > for (j in 1:177) {
> > for (k in 182) { #i,j and k values are choosen
> > with regad that K cannot exceed seasonality period/2
> > z1 < fourier(ts(demand,frequency = 7), K=i)
> > z2 < fourier(ts(demand,frequency=354), K=j)
> > z3 < fourier(ts(demand,frequency = 365),K=k)
> > fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> > seasonal = FALSE)
> > fit$aicc
> > AICc[,1]<i
> > AICc[,2]<j
> > AICc[,3]<k
> > AICc[,4]<fit$aicc
> > }
> >change
> > }
> > }
> > AICc
On Thu, 7 Feb 2019 at 13:44, salah maadawy < [hidden email]> wrote:
>
> Hi Micheal, Maybe there is a simple way but i wanted to get the lowest aicc
> ana i could not find a way to do so, that's why i created the table to
> store all possible outcomes and then i can easily get the minimum value and
> the values of (i,j and k) used for that minimum value. The first column in
> the table is AICc[,1] to store i and second column for j and so on. Maybe i
> am mistaken and this won't give me what i want, the code been running for 5
> hours now. So i am waiting
>
> On Wed, Feb 6, 2019 at 4:59 PM Michael Dewey < [hidden email]>
> wrote:
>
> > This is not an answer to your speed problem but are your assignments to
> > AICc[,1] and so on doing what you hope they are doing?
> >
> > Michael
> >
> > On 06/02/2019 12:03, salah maadawy wrote:
> > > i am a beginner regarding R but i am trying to do a simple thing, but it
> > is
> > > taking too much time and i am asking if there is any way to achieve what
> > i
> > > need, i have a time series data set with 730 data points, i detected 7,
> > 354
> > > and 365 seasonality periods. i am trying to use Fourier terms for
> > > seasonality and for loop to get the K value for each while minimizing
> > AICc,
> > > my code is
> > >
> > > AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
> > > for (j in 1:177) {
> > > for (k in 182) { #i,j and k values are choosen
> > > with regad that K cannot exceed seasonality period/2
> > > z1 < fourier(ts(demand,frequency = 7), K=i)
> > > z2 < fourier(ts(demand,frequency=354), K=j)
> > > z3 < fourier(ts(demand,frequency = 365),K=k)
> > > fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> > > seasonal = FALSE)
> > > fit$aicc
> > > AICc[,1]<i
> > > AICc[,2]<j
> > > AICc[,3]<k
> > > AICc[,4]<fit$aicc
> > > }
> > >
> > > }
> > > }
> > > AICc
> > >
> > > i have created a data table to store AICc values from all possible i,j,k
> > > combinations so that i can find later the minimum AICc value. the problem
> > > now is that it is taking forever to do so not only to iterate all
> > > combinations but also due to the large K values.
> > >
> > > , is there any possible solution for this? thank you in advance
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > ______________________________________________
> > > [hidden email] mailing list  To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/rhelp> > > PLEASE do read the posting guide
> > http://www.Rproject.org/postingguide.html> > > and provide commented, minimal, selfcontained, reproducible code.
> > >
> >
> > 
> > Michael
> > http://www.dewey.myzen.co.uk/home.html> >
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list  To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/rhelp> PLEASE do read the posting guide http://www.Rproject.org/postingguide.html> and provide commented, minimal, selfcontained, reproducible code.

Bob O'Hara
Institutt for matematiske fag
NTNU
7491 Trondheim
Norway
Mobile: +47 915 54 416
Journal of Negative Results  EEB: www.jnreeb.org
______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.


thank you all for your suggestions and feedback, with further search and
experimentation, the problem is with the auto.arima function with large k
values, it takes 4 min to compute one model with k=30 and the time
increases with K, so i used your suggestions for collecting the output but
i limited my loop (i=3,j=25 and k=25) , it took around 17 hours to finish
already (1875 models). thank you again.
On Thu, Feb 7, 2019 at 4:10 PM Bob O'Hara < [hidden email]> wrote:
> With 96k model fits it's going to be slow, so you might want to think
> first about whether you need to do them all. Beyond that, I think this
> is more in the R style, so might be quicker (I don't know how much the
> loops are slowing you down), and even if not it should be easier to
> adapt.
>
> The other thing to think about is parallelising the code  the
> parallel package should help.
>
> FitModel < function(K, data) {
> z1 < fourier(ts(data,frequency = 7), K=K["i"])
> z2 < fourier(ts(data,frequency=354), K=K["j"])
> z3 < fourier(ts(data,,frequency = 365),K=K["k"])
> fit < auto.arima(data,, xreg =cbind(z1,z2,z3), seasonal = FALSE)
> fit$aicc
> }
>
> # smaller MaxOrders used so if you run it like this, it won't take 5 hours
> MaxOrders < expand.grid(i = 1:3, j=1=7, k=1:8)
> AICc < apply(MaxOrders, FitModel, data=demand)
>
> Bob
>
> AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3) {
> > > for (j in 1:177) {
> > > for (k in 182) { #i,j and k values are choosen
> > > with regad that K cannot exceed seasonality period/2
> > > z1 < fourier(ts(demand,frequency = 7), K=i)
> > > z2 < fourier(ts(demand,frequency=354), K=j)
> > > z3 < fourier(ts(demand,frequency = 365),K=k)
> > > fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> > > seasonal = FALSE)
> > > fit$aicc
> > > AICc[,1]<i
> > > AICc[,2]<j
> > > AICc[,3]<k
> > > AICc[,4]<fit$aicc
> > > }
> > >change
> > > }
> > > }
> > > AICc
>
> On Thu, 7 Feb 2019 at 13:44, salah maadawy < [hidden email]> wrote:
> >
> > Hi Micheal, Maybe there is a simple way but i wanted to get the lowest
> aicc
> > ana i could not find a way to do so, that's why i created the table to
> > store all possible outcomes and then i can easily get the minimum value
> and
> > the values of (i,j and k) used for that minimum value. The first column
> in
> > the table is AICc[,1] to store i and second column for j and so on.
> Maybe i
> > am mistaken and this won't give me what i want, the code been running
> for 5
> > hours now. So i am waiting
> >
> > On Wed, Feb 6, 2019 at 4:59 PM Michael Dewey < [hidden email]>
> > wrote:
> >
> > > This is not an answer to your speed problem but are your assignments to
> > > AICc[,1] and so on doing what you hope they are doing?
> > >
> > > Michael
> > >
> > > On 06/02/2019 12:03, salah maadawy wrote:
> > > > i am a beginner regarding R but i am trying to do a simple thing,
> but it
> > > is
> > > > taking too much time and i am asking if there is any way to achieve
> what
> > > i
> > > > need, i have a time series data set with 730 data points, i detected
> 7,
> > > 354
> > > > and 365 seasonality periods. i am trying to use Fourier terms for
> > > > seasonality and for loop to get the K value for each while minimizing
> > > AICc,
> > > > my code is
> > > >
> > > > AICc< data.table(matrix(nrow = 96642, ncol = 4))for (i in 1:3)
> {
> > > > for (j in 1:177) {
> > > > for (k in 182) { #i,j and k values are
> choosen
> > > > with regad that K cannot exceed seasonality period/2
> > > > z1 < fourier(ts(demand,frequency = 7), K=i)
> > > > z2 < fourier(ts(demand,frequency=354), K=j)
> > > > z3 < fourier(ts(demand,frequency = 365),K=k)
> > > > fit < auto.arima(demand, xreg =cbind(z1,z2,z3),
> > > > seasonal = FALSE)
> > > > fit$aicc
> > > > AICc[,1]<i
> > > > AICc[,2]<j
> > > > AICc[,3]<k
> > > > AICc[,4]<fit$aicc
> > > > }
> > > >
> > > > }
> > > > }
> > > > AICc
> > > >
> > > > i have created a data table to store AICc values from all possible
> i,j,k
> > > > combinations so that i can find later the minimum AICc value. the
> problem
> > > > now is that it is taking forever to do so not only to iterate all
> > > > combinations but also due to the large K values.
> > > >
> > > > , is there any possible solution for this? thank you in advance
> > > >
> > > > [[alternative HTML version deleted]]
> > > >
> > > > ______________________________________________
> > > > [hidden email] mailing list  To UNSUBSCRIBE and more, see
> > > > https://stat.ethz.ch/mailman/listinfo/rhelp> > > > PLEASE do read the posting guide
> > > http://www.Rproject.org/postingguide.html> > > > and provide commented, minimal, selfcontained, reproducible code.
> > > >
> > >
> > > 
> > > Michael
> > > http://www.dewey.myzen.co.uk/home.html> > >
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list  To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/rhelp> > PLEASE do read the posting guide
> http://www.Rproject.org/postingguide.html> > and provide commented, minimal, selfcontained, reproducible code.
>
>
>
> 
> Bob O'Hara
> Institutt for matematiske fag
> NTNU
> 7491 Trondheim
> Norway
>
> Mobile: +47 915 54 416
> Journal of Negative Results  EEB: www.jnreeb.org
>
[[alternative HTML version deleted]]
______________________________________________
[hidden email] mailing list  To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/rhelpPLEASE do read the posting guide http://www.Rproject.org/postingguide.htmland provide commented, minimal, selfcontained, reproducible code.

