newbie: fourier series for time series data

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

newbie: fourier series for time series data

eddie smith
Hi Guys,

I had a monthly time series's data of land temperature from 1980 to 2008.
After plotting a scatter diagram, it seems that annually, there is a semi
sinusoidal cycle. How do I run Fourier's series to the data so that I can
fit model on it?

I am really sorry for my question sound stupid, but I just don't know where
to start.

I am desperately looking for help from you guys.

Thanks in advance.

Eddie

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: newbie: fourier series for time series data

Spencer Graves-2
On 5/31/2011 5:12 AM, eddie smith wrote:
> Hi Guys,
>
> I had a monthly time series's data of land temperature from 1980 to 2008.
> After plotting a scatter diagram, it seems that annually, there is a semi
> sinusoidal cycle. How do I run Fourier's series to the data so that I can
> fit model on it?

       There are several methods.


             1.  The simplest would be to select the number of terms you
want, put the data into a data.frame, and use lm(y ~ sin(t/period) +
cos(t/period) + sin(2*t/period) + cos(2*t/period) + ..., data),
including as many terms as you want in the series.  This is not
recommended, because it ignores the time series effects and does not
apply a smoothness penalty to the Fourier approximation.


             2.  A second is to use the 'fda' package.  Examples are
provided (even indexed) in Ramsay, Hooker and Graves (2009) Functional
Data Analysis with R and Matlab (Springer).  This is probably what
Ramsay and Hooker would do, but I wouldn't, because it doesn't treat the
time series as a time series.  It also requires more work on your part.


             3.  A third general class of approaches uses Kalman
filtering, also called dynamic linear models or state space models.  
This would allow you to estimate a differential equation model, whose
solution could be a damped sinusoid.  It would also allow you to
estimate regression coefficients of a finite Fourier series but without
the smoothness penalty you would get with 'fda'.  For this, I recommend
the 'dlm' package with its vignette and companion book, Petris, Petrone
and Campagnoli (2009) Dynamic Linear Models with R (Springer).


       If you want something quick and dirty, you might want option 1.  
For that, I might use option 2, because I know and understand it
moderately well (being third author on the book).  However, if you
really want to understand time series, I recommend option 3.  That has
the additional advantage that I think it would have the greatest chances
of acceptance in a refereed academic journal of the three approaches.


> I am really sorry for my question sound stupid, but I just don't know where
> to start.

       There also are specialized email lists that you might consider
for a future post.  Go to www.r-project.org <http://www.r-project.org/>
-> "Mailing Lists".  In particular, you might be most interested in
R-sig-ecology.


       Hope this helps.
       Spencer Graves

> I am desperately looking for help from you guys.
>
> Thanks in advance.
>
> Eddie
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email]  mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guidehttp://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

--
Spencer Graves, PE, PhD
President and Chief Operating Officer
Structure Inspection and Monitoring, Inc.
751 Emerson Ct.
San José, CA 95126
ph:  408-655-4567


        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: newbie: fourier series for time series data

Oliver-3
In reply to this post by eddie smith
To get an overview, you could just play around
with fft() and spectrum(), two functions, which
R offers by default (assuming you know Fourier Transformation).
fft() gives you back complex spectrum.


There also is a refcard that gives an overview on Time Series analysis
from Vito Ricci:

  http://cran.r-project.org/doc/contrib/Ricci-refcard-ts.pdf


And there also is a Time series Analysis package (TSA-package):
  which offers a lot of functions:

  http://cran.r-project.org/web/packages/TSA/index.html

In one of the last issues of R-Journal I remember, that there also was something
written about decomposition.

Ciao,
   Oliver

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: newbie: fourier series for time series data

Mike Marchywka
In reply to this post by Spencer Graves-2

( hotmail won't mark text so I'm top posting... )
 
Can you post the data? Personally I'd just plot abs(fft(x))
and see what you see, as well as looking at Im(fft(x))/Re(fft(x))
or phase spectrum. Now, presumably you are nominally looking
for something with a period of 1 year, that part could
be expressed in harmonic specrum as suggested below, but you'd
also be looking for trends and noises of various types- additive
gausssian, amplitude modulation, maybe even frequncy modulation, etc.
I guess you could remove a few power terms ( average, linear, etc)
just to simplify ( often you get this spectrum with huge uniformative
DC or zero frequency component just gets in the way). You can probably
find book online dealing with signal processing and RF ( this is where
I'm used to seeing this things). It would of course be helpful to
then examine known simple cases and see if you can tell them apart.
Create fft( sin(t)*(1+a*sin(epsilon*t)+b*t ) for example.


I guess if you want to look at writing a model, you could look at
phase portrait ( plot derviative versus value ) to again get some idea what
you may have that makes sense as model to fit.



----------------------------------------
Date: Tue, 31 May 2011 10:35:16 -0700
From: [hidden email]
To: [hidden email]
CC: [hidden email]
Subject: Re: [R] newbie: fourier series for time series data


On 5/31/2011 5:12 AM, eddie smith wrote:
> Hi Guys,
>
> I had a monthly time series's data of land temperature from 1980 to 2008.
> After plotting a scatter diagram, it seems that annually, there is a semi
> sinusoidal cycle. How do I run Fourier's series to the data so that I can
> fit model on it?

There are several methods.


1. The simplest would be to select the number of terms you
want, put the data into a data.frame, and use lm(y ~ sin(t/period) +
cos(t/period) + sin(2*t/period) + cos(2*t/period) + ..., data),
including as many terms as you want in the series. This is not
recommended, because it ignores the time series effects and does not
apply a smoothness penalty to the Fourier approximation.


2. A second is to use the 'fda' package. Examples are
provided (even indexed) in Ramsay, Hooker and Graves (2009) Functional
Data Analysis with R and Matlab (Springer). This is probably what
Ramsay and Hooker would do, but I wouldn't, because it doesn't treat the
time series as a time series. It also requires more work on your part.


3. A third general class of approaches uses Kalman
filtering, also called dynamic linear models or state space models.
This would allow you to estimate a differential equation model, whose
solution could be a damped sinusoid. It would also allow you to
estimate regression coefficients of a finite Fourier series but without
the smoothness penalty you would get with 'fda'. For this, I recommend
the 'dlm' package with its vignette and companion book, Petris, Petrone
and Campagnoli (2009) Dynamic Linear Models with R (Springer).


If you want something quick and dirty, you might want option 1.
For that, I might use option 2, because I know and understand it
moderately well (being third author on the book). However, if you
really want to understand time series, I recommend option 3. That has
the additional advantage that I think it would have the greatest chances
of acceptance in a refereed academic journal of the three approaches.


> I am really sorry for my question sound stupid, but I just don't know where
> to start.

There also are specialized email lists that you might consider
for a future post. Go to www.r-project.org
-> "Mailing Lists". In particular, you might be most interested in
R-sig-ecology.


Hope this helps.
Spencer Graves

> I am desperately looking for help from you guys.
>
> Thanks in advance.
>
> Eddie
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guidehttp://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


--
Spencer Graves, PE, PhD
President and Chief Operating Officer
Structure Inspection and Monitoring, Inc.
751 Emerson Ct.
San José, CA 95126
ph: 408-655-4567


[[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.    
______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: newbie: fourier series for time series data

eddie smith
Hi Spencer and Mike,

Thank you very much! I never expect to get 3 replies in one day. You all are
really helpful. While I am struggling reading all the documents suggested,
here is the data. Maybe this will help to clear up what I am intending to
do.

Thank in advance guys.

Eddie

On Tue, May 31, 2011 at 10:39 PM, Mike Marchywka <[hidden email]>wrote:

>
> ( hotmail won't mark text so I'm top posting... )
>
> Can you post the data? Personally I'd just plot abs(fft(x))
> and see what you see, as well as looking at Im(fft(x))/Re(fft(x))
> or phase spectrum. Now, presumably you are nominally looking
> for something with a period of 1 year, that part could
> be expressed in harmonic specrum as suggested below, but you'd
> also be looking for trends and noises of various types- additive
> gausssian, amplitude modulation, maybe even frequncy modulation, etc.
> I guess you could remove a few power terms ( average, linear, etc)
> just to simplify ( often you get this spectrum with huge uniformative
> DC or zero frequency component just gets in the way). You can probably
> find book online dealing with signal processing and RF ( this is where
> I'm used to seeing this things). It would of course be helpful to
> then examine known simple cases and see if you can tell them apart.
> Create fft( sin(t)*(1+a*sin(epsilon*t)+b*t ) for example.
>
>
> I guess if you want to look at writing a model, you could look at
> phase portrait ( plot derviative versus value ) to again get some idea what
> you may have that makes sense as model to fit.
>
>
>
> ----------------------------------------
> Date: Tue, 31 May 2011 10:35:16 -0700
> From: [hidden email]
> To: [hidden email]
> CC: [hidden email]
> Subject: Re: [R] newbie: fourier series for time series data
>
>
> On 5/31/2011 5:12 AM, eddie smith wrote:
> > Hi Guys,
> >
> > I had a monthly time series's data of land temperature from 1980 to 2008.
> > After plotting a scatter diagram, it seems that annually, there is a semi
> > sinusoidal cycle. How do I run Fourier's series to the data so that I can
> > fit model on it?
>
> There are several methods.
>
>
> 1. The simplest would be to select the number of terms you
> want, put the data into a data.frame, and use lm(y ~ sin(t/period) +
> cos(t/period) + sin(2*t/period) + cos(2*t/period) + ..., data),
> including as many terms as you want in the series. This is not
> recommended, because it ignores the time series effects and does not
> apply a smoothness penalty to the Fourier approximation.
>
>
> 2. A second is to use the 'fda' package. Examples are
> provided (even indexed) in Ramsay, Hooker and Graves (2009) Functional
> Data Analysis with R and Matlab (Springer). This is probably what
> Ramsay and Hooker would do, but I wouldn't, because it doesn't treat the
> time series as a time series. It also requires more work on your part.
>
>
> 3. A third general class of approaches uses Kalman
> filtering, also called dynamic linear models or state space models.
> This would allow you to estimate a differential equation model, whose
> solution could be a damped sinusoid. It would also allow you to
> estimate regression coefficients of a finite Fourier series but without
> the smoothness penalty you would get with 'fda'. For this, I recommend
> the 'dlm' package with its vignette and companion book, Petris, Petrone
> and Campagnoli (2009) Dynamic Linear Models with R (Springer).
>
>
> If you want something quick and dirty, you might want option 1.
> For that, I might use option 2, because I know and understand it
> moderately well (being third author on the book). However, if you
> really want to understand time series, I recommend option 3. That has
> the additional advantage that I think it would have the greatest chances
> of acceptance in a refereed academic journal of the three approaches.
>
>
> > I am really sorry for my question sound stupid, but I just don't know
> where
> > to start.
>
> There also are specialized email lists that you might consider
> for a future post. Go to www.r-project.org
> -> "Mailing Lists". In particular, you might be most interested in
> R-sig-ecology.
>
>
> Hope this helps.
> Spencer Graves
>
> > I am desperately looking for help from you guys.
> >
> > Thanks in advance.
> >
> > Eddie
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guidehttp://
> www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>
> --
> Spencer Graves, PE, PhD
> President and Chief Operating Officer
> Structure Inspection and Monitoring, Inc.
> 751 Emerson Ct.
> San José, CA 95126
> ph: 408-655-4567
>
>
> [[alternative HTML version deleted]]
>
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

landtemp.txt (2K) Download Attachment