[R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

dave fournier

There has recently been some discussion on the list about
AD Model builder and the suitability of R for constructing the
types of models used in fisheries management.

   https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html

   https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html

I  think that many R users understimate the numerical challenges
that some of the typical nonlinear statistical model used in different
fields present. R may not be a suitable platform for development for
such models.

Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
with Canadian federal fisheries undertook an investigation
comparing various statistical modeling packages for a simple
age-structured statistical model of the type commonly used in
fisheries. They compared AD Mdel Builder, Gauss, Matlab, and
Splus. Unfortunately a working model could not be produced with Splus
so its times could not be included in the comparison. It is possible
to produce a working model with the present day version of R so that
R can now be directly compared with AD Model Builder for this type of model.

I have put the results of the test together with the original
Schnute and Richards paper and the working R and AD Model Builder
codes on Otter's web site

     http://otter-rsch.ca/tresults.htm

The results are that AD Model builder is roughly 1000 times faster than
R for this problem. ADMB takes about 2 seconds to converge while
R takes over 90 minutes.

This is a simple toy example. Real fisheries models are often hundred of
times more computationally intensive as this one.

        Cheers,

         Dave
~
--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

Mike Prager
dave fournier <[hidden email]> wrote:

> I  think that many R users understimate the numerical challenges
> that some of the typical nonlinear statistical model used in different
> fields present. R may not be a suitable platform for development for
> such models.
>
> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
> with Canadian federal fisheries undertook an investigation
> comparing various statistical modeling packages for a simple
> age-structured statistical model of the type commonly used in
> fisheries. [...] It is possible
> to produce a working model with the present day version of R so that
> R can now be directly compared with AD Model Builder for this type of model.
>
> The results are that AD Model builder is roughly 1000 times faster than
> R for this problem. ADMB takes about 2 seconds to converge while
> R takes over 90 minutes.

Our group's experiences reflect, at least qualitatively, what
Dave says above.  We use R for analyzing results from models
written in his AD Model Builder, and a couple of years ago, we
started programming one of our models directly in R.  We quickly
abandoned that idea because of lengthy execution time under R.
That is not a judgement of either piece of software.  R and ADMB
are designed for different types of task, and it seems to me
that they complement each other well.

That experience was in part the genesis of our X2R software (now
at CRAN -- pardon the plug), which saves results from ADMB
models into a format that R can read as a list.  We feel that
now we have the best of both worlds -- fast execution with ADMB,
followed by the programming ease and excellent graphics of R for
analysis of results and projections under numerous scenarios.

--
Mike Prager, NOAA, Beaufort, NC
* Opinions expressed are personal and not represented otherwise.
* Any use of tradenames does not constitute a NOAA endorsement.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

Tony Plate-3
In reply to this post by dave fournier
Did you try supplying gradient information to nlminb?  (I note that
nlminb is used for the optimization, but I don't see any gradient
information supplied to it.) I would suspect that supplying gradient
information would greatly speed up the computation (as you note in
comments at http://otter-rsch.ca/tresults.htm.)

I'm curious -- when you say "R may not be a suitable platform for
development for such models", what aspect of R do you feel is lacking?
Is it the specific optimization routines available, or is it some other
more general aspect?

Also, another optimization algorithm available in R is the "L-BFGS-B"
method for optim() in the MASS package.  I've had extremely good
experiences with using this code in S-PLUS.  It can take box
constraints, and can use gradient information.  It is my first choice
for most optimization problems, and I believe it is very widely used.
Did you try using that optimization routine with this problem?

-- Tony Plate

dave fournier wrote:

> There has recently been some discussion on the list about
> AD Model builder and the suitability of R for constructing the
> types of models used in fisheries management.
>
>    https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html
>
>    https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html
>
> I  think that many R users understimate the numerical challenges
> that some of the typical nonlinear statistical model used in different
> fields present. R may not be a suitable platform for development for
> such models.
>
> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
> with Canadian federal fisheries undertook an investigation
> comparing various statistical modeling packages for a simple
> age-structured statistical model of the type commonly used in
> fisheries. They compared AD Mdel Builder, Gauss, Matlab, and
> Splus. Unfortunately a working model could not be produced with Splus
> so its times could not be included in the comparison. It is possible
> to produce a working model with the present day version of R so that
> R can now be directly compared with AD Model Builder for this type of model.
>
> I have put the results of the test together with the original
> Schnute and Richards paper and the working R and AD Model Builder
> codes on Otter's web site
>
>      http://otter-rsch.ca/tresults.htm
>
> The results are that AD Model builder is roughly 1000 times faster than
> R for this problem. ADMB takes about 2 seconds to converge while
> R takes over 90 minutes.
>
> This is a simple toy example. Real fisheries models are often hundred of
> times more computationally intensive as this one.
>
>         Cheers,
>
>          Dave
> ~

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

dave fournier
In reply to this post by dave fournier


       Dave
 > Did you try supplying gradient information to nlminb?  (I note that
nlminb is used for the optimization, but I don't see any gradient
information supplied to it.) I would suspect that supplying gradient
information would greatly speed up the computation (as you note in
comments at http://otter-rsch.ca/tresults.htm.)

Actually you should probably ask Norm Olsen these questions.
I am not proficient in R and am just using his code.

However I can say that providing derivatives for such a model is a
highly nontrivial exercise. As I said in my posting, the  R script and
data are available to anyone who feels that the exercise was not carried
out properly and would like to improve on it. Also one does not need
to provide derivatives to the AD Model Builder program.

Finally suppose that you are very good at calculating derivatives and
manage to get them right. Then someone else comes along who wants to
modify the model. Unless they are also very good at calculating
derivatives there will be trouble.

 >
 > I'm curious -- when you say "R may not be a suitable platform for
development for such models", what aspect of R do you feel is lacking?
Is it the specific optimization routines available, or is it some other
more general aspect?

2 seconds vs 90 minutes. For a real problem of tihs type the timings
would probably be something like 10 minutes vs more than 2,700 minutes.

 >
 > Also, another optimization algorithm available in R is the "L-BFGS-B"
method for optim() in the MASS package.  I've had extremely good
experiences with using this code in S-PLUS.  It can take box
constraints, and can use gradient information.  It is my first choice
for most optimization problems, and I believe it is very widely used.
Did you try using that optimization routine with this problem?
 >
 > -- Tony Plate
 >
 > dave fournier wrote:
 >> There has recently been some discussion on the list about
 >> AD Model builder and the suitability of R for constructing the
 >> types of models used in fisheries management.
 >>
 >>    https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html
 >>
 >>    https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html
 >>
 >> I  think that many R users understimate the numerical challenges
 >> that some of the typical nonlinear statistical model used in different
 >> fields present. R may not be a suitable platform for development for
 >> such models.
 >>
 >> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
 >> with Canadian federal fisheries undertook an investigation
 >> comparing various statistical modeling packages for a simple
 >> age-structured statistical model of the type commonly used in
 >> fisheries. They compared AD Mdel Builder, Gauss, Matlab, and
 >> Splus. Unfortunately a working model could not be produced with Splus
 >> so its times could not be included in the comparison. It is possible
 >> to produce a working model with the present day version of R so that
 >> R can now be directly compared with AD Model Builder for this type
of model.
 >>
 >> I have put the results of the test together with the original
 >> Schnute and Richards paper and the working R and AD Model Builder
 >> codes on Otter's web site
 >>
 >>      http://otter-rsch.ca/tresults.htm
 >>
 >> The results are that AD Model builder is roughly 1000 times faster than
 >> R for this problem. ADMB takes about 2 seconds to converge while
 >> R takes over 90 minutes.
 >>
 >> This is a simple toy example. Real fisheries models are often hundred of
 >> times more computationally intensive as this one.
 >>
 >>         Cheers,
 >>
 >>          Dave
 >> ~
 >


--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com

--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

Spencer Graves
In reply to this post by Mike Prager
Hi, Mike & Dave:

      Have you considered nonlinear mixed effects models for the types
of problems considered in the comparison paper you cite?  Those
"benchmark trials" consider "T years of data ... for A age classes and
the total number of parameters is m = T+A+5".  Without knowing more
about the problem, I suspect that the T year parameters and the A age
class parameters might be better modeled as random effects.  If this
were done, the optimization problem would then involve 7 parameters, the
5 fixed-effect parameters suggested by the computation of "m" plus two
variance parameters, one for the random "year" effects and another for
the random "age class" effect.  This would replace the problem of
maximizing, e.g., a likelihood over T+A+5 parameters with one of
maximizing a marginal likelihood over 2+5 parameters after integrating
out the T and A random effects.

      These integrations may not be easy, and I might stick with the
fixed-effects solution if I couldn't get answers in the available time
using a model I thought would be theoretically more appropriate.  Also,
I might use the fixed-effects solution to get starting values for an
attempt to maximize a more appropriate marginal likelihood.  For the
latter, I might first try 'nlmle'.  If that failed, I might explore
Markov Chain Monte Carlo (MCMC).  I have not done MCMC myself, but the
"MCMCpack" R package looks like it might make it feasible for the types
of problems considered in this comparison.  The CRAN summary of that
package led me to an Adobe Acrobat version of a PPT slide presentation
that seemed to consider just this type of problem (e.g.,
http://mcmcpack.wustl.edu/files/MartinQuinnMCMCpackslides.pdf).

      Have you considered that?
      Hope this helps.
      Spencer Graves

Mike Prager wrote:

> dave fournier <[hidden email]> wrote:
>
>  
>> I  think that many R users understimate the numerical challenges
>> that some of the typical nonlinear statistical model used in different
>> fields present. R may not be a suitable platform for development for
>> such models.
>>
>> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
>> with Canadian federal fisheries undertook an investigation
>> comparing various statistical modeling packages for a simple
>> age-structured statistical model of the type commonly used in
>> fisheries. [...] It is possible
>> to produce a working model with the present day version of R so that
>> R can now be directly compared with AD Model Builder for this type of model.
>>
>> The results are that AD Model builder is roughly 1000 times faster than
>> R for this problem. ADMB takes about 2 seconds to converge while
>> R takes over 90 minutes.
>>    
>
> Our group's experiences reflect, at least qualitatively, what
> Dave says above.  We use R for analyzing results from models
> written in his AD Model Builder, and a couple of years ago, we
> started programming one of our models directly in R.  We quickly
> abandoned that idea because of lengthy execution time under R.
> That is not a judgement of either piece of software.  R and ADMB
> are designed for different types of task, and it seems to me
> that they complement each other well.
>
> That experience was in part the genesis of our X2R software (now
> at CRAN -- pardon the plug), which saves results from ADMB
> models into a format that R can read as a list.  We feel that
> now we have the best of both worlds -- fast execution with ADMB,
> followed by the programming ease and excellent graphics of R for
> analysis of results and projections under numerous scenarios.
>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

Douglas Bates-2
In reply to this post by dave fournier
On 11/24/06, dave fournier <[hidden email]> wrote:
>
>
>        Dave
>  > Did you try supplying gradient information to nlminb?  (I note that
> nlminb is used for the optimization, but I don't see any gradient
> information supplied to it.) I would suspect that supplying gradient
> information would greatly speed up the computation (as you note in
> comments at http://otter-rsch.ca/tresults.htm.)

> Actually you should probably ask Norm Olsen these questions.
> I am not proficient in R and am just using his code.

Don't you find it somewhat disingenuous that you publish a comparison
between the AD Model Builder software that you sell and R - a
comparison that shows a tremendous advantage for your software - and
then you write "I am not proficient in R"?

Had you been proficient in R you might have known about the symbolic
differentiation capabilities, specifically the deriv function, that
have been part of the S language since the late 1980s.  I believe that
the 'AD' in "AD model builder" stands for automatic differentiation,
which is actually something that John Chambers and I discussed at
length when we were developing nonlinear modeling methods for S.  In
the end we went with symbolic differentiation rather than automatic
differentiation because we felt that symbolic was more flexible.

This is not to say that automatic differentiation isn't a perfectly
legitimate technique.  However, my recollection is that it would have
required extensive revisions to the arithmetic expression evaluator,
which is already very tricky code because of the "recycling rule" and
the desire to shield users from knowledge of the internal
representations and such details as whether you are using logical or
integer or double precision operands or a combination.

If you want to see these details you can, of course, examine the
source code.  I don't believe we would have the opportunity to examine
how you implemented automatic differentiation.

I must also agree with Spencer Graves that when I start reading a
description of a nonlinear model with over 100 parameters, the example
that you chose, I immediately start thinking of nonlinear mixed
effects models.  In my experience the only way in which a nonlinear
model ends up with that number of parameters is through applying an
underlying model with a low number of parameters to various groups
within the data.  Table 2 in the Schnute et al. paper to which you
make reference states that the number of parameters in the model is T
+ A + 5 where T is the number of years of data and A is the number of
age classes.  To me that looks a lot like a nonlinear mixed effects
model.

Also, your choice of subject heading for your message seems
deliberatively provocative.  You seem to be implying that you are
discussing a comparisons of AD Model Builder and R on all aspects of
nonlinear statistical modeling but you are only discussing one
comparison on simulated data using a model from the applications area
for which you wrote AD Model Builder.  Then you follow up by saying "I
am not proficient in R" and your results for R are from applying code
that someone else gave you.

It seems that ADMB had a bit of a "home-field advantage" in this
particular comparison.

I view nonlinear statistical modeling differently.  I have had a bit
of experience in the area and I find that I want to examine the data
carefully, usually through plots, before I embark on fitting
complicated models.  I like to have some assurance that the model
makes sense in the context of the data.  (In your example you don't
need to worry about appropriateness of the model because the data were
simulated.) I would never try to fit a nonlinear model with 100
parameters to data without carefully examining the data, and
especially selected subsets of the data, first.  For this the
flexibility of the S language and tools like lattice graphics that
were developed in this language are invaluable to me.  The flexibility
of data manipulation and graphics for interactive exploration of data
is what attracted me to S in the first place.

I realize that for many people the area of nonlinear statistical
modeling is reduced to "Fit this model to these data and don't ask any
questions.   Just give me parameter estimates and p-values."  If that
is your situation then it would make sense to use software that gets
you those estimates as quickly as possible with a minimum of effort.
I'm just happy that I get to turn down people who ask me to do that.
I like that fact that I can spend my time asking questions about the
data and of the data.


> However I can say that providing derivatives for such a model is a
> highly nontrivial exercise. As I said in my posting, the  R script and
> data are available to anyone who feels that the exercise was not carried
> out properly and would like to improve on it. Also one does not need
> to provide derivatives to the AD Model Builder program.
>
> Finally suppose that you are very good at calculating derivatives and
> manage to get them right. Then someone else comes along who wants to
> modify the model. Unless they are also very good at calculating
> derivatives there will be trouble.
>
>  >
>  > I'm curious -- when you say "R may not be a suitable platform for
> development for such models", what aspect of R do you feel is lacking?
> Is it the specific optimization routines available, or is it some other
> more general aspect?
>
> 2 seconds vs 90 minutes. For a real problem of tihs type the timings
> would probably be something like 10 minutes vs more than 2,700 minutes.
>
>  >
>  > Also, another optimization algorithm available in R is the "L-BFGS-B"
> method for optim() in the MASS package.  I've had extremely good
> experiences with using this code in S-PLUS.  It can take box
> constraints, and can use gradient information.  It is my first choice
> for most optimization problems, and I believe it is very widely used.
> Did you try using that optimization routine with this problem?
>  >
>  > -- Tony Plate
>  >
>  > dave fournier wrote:
>  >> There has recently been some discussion on the list about
>  >> AD Model builder and the suitability of R for constructing the
>  >> types of models used in fisheries management.
>  >>
>  >>    https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html
>  >>
>  >>    https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html
>  >>
>  >> I  think that many R users understimate the numerical challenges
>  >> that some of the typical nonlinear statistical model used in different
>  >> fields present. R may not be a suitable platform for development for
>  >> such models.
>  >>
>  >> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
>  >> with Canadian federal fisheries undertook an investigation
>  >> comparing various statistical modeling packages for a simple
>  >> age-structured statistical model of the type commonly used in
>  >> fisheries. They compared AD Mdel Builder, Gauss, Matlab, and
>  >> Splus. Unfortunately a working model could not be produced with Splus
>  >> so its times could not be included in the comparison. It is possible
>  >> to produce a working model with the present day version of R so that
>  >> R can now be directly compared with AD Model Builder for this type
> of model.
>  >>
>  >> I have put the results of the test together with the original
>  >> Schnute and Richards paper and the working R and AD Model Builder
>  >> codes on Otter's web site
>  >>
>  >>      http://otter-rsch.ca/tresults.htm
>  >>
>  >> The results are that AD Model builder is roughly 1000 times faster than
>  >> R for this problem. ADMB takes about 2 seconds to converge while
>  >> R takes over 90 minutes.
>  >>
>  >> This is a simple toy example. Real fisheries models are often hundred of
>  >> times more computationally intensive as this one.
>  >>
>  >>         Cheers,
>  >>
>  >>          Dave
>  >> ~
>  >
>
>
> --
> David A. Fournier
> P.O. Box 2040,
> Sidney, B.C. V8l 3S3
> Canada
> Phone/FAX 250-655-3364
> http://otter-rsch.com
>
> --
> David A. Fournier
> P.O. Box 2040,
> Sidney, B.C. V8l 3S3
> Canada
> Phone/FAX 250-655-3364
> http://otter-rsch.com
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

H. Skaug
In reply to this post by Spencer Graves
Spencer,

I tried the mixed effects approach you suggest using the random effects
module of
AD Model Builder: (http://www.otter-rsch.ca/admbre/admbre.html). What are
94 unbounded parameters in Schnute et al (1998), now become realizations
of a Gaussian random variable, with the corresponding standard deviation
being
estimated as a parameter. The approach works, but the computation time is
increased substantially. This is however  understandable
as the computational problem is a very different one. The likelihood
function
now involves an integral in dimension 94, which I believe cannot be broken
into
a product of lower dimensional integrals as is usual for clustered data (the
reason being the recursive nature of the population dynamics).

hans

_______________________________



Spencer Graves wrote:

>     Have you considered nonlinear mixed effects models for the types
>of problems considered in the comparison paper you cite?  Those
>"benchmark trials" consider "T years of data ... for A age classes and
>the total number of parameters is m = T+A+5".  Without knowing more
>about the problem, I suspect that the T year parameters and the A age
>class parameters might be better modeled as random effects.  If this
>were done, the optimization problem would then involve 7 parameters, the
>5 fixed-effect parameters suggested by the computation of "m" plus two
>variance parameters, one for the random "year" effects and another for
>the random "age class" effect.  This would replace the problem of
>maximizing, e.g., a likelihood over T+A+5 parameters with one of
>maximizing a marginal likelihood over 2+5 parameters after integrating
>out the T and A random effects.
>
>     These integrations may not be easy, and I might stick with the
>fixed-effects solution if I couldn't get answers in the available time
>using a model I thought would be theoretically more appropriate.  Also,
>I might use the fixed-effects solution to get starting values for an
>attempt to maximize a more appropriate marginal likelihood.  For the
>latter, I might first try 'nlmle'.  If that failed, I might explore
>Markov Chain Monte Carlo (MCMC).  I have not done MCMC myself, but the
>"MCMCpack" R package looks like it might make it feasible for the types
>of problems considered in this comparison.  The CRAN summary of that
>package led me to an Adobe Acrobat version of a PPT slide presentation
>that seemed to consider just this type of problem (e.g.,
> http://mcmcpack.wustl.edu/files/MartinQuinnMCMCpackslides.pdf).
>
>     Have you considered that?
>     Hope this helps.
>     Spencer Graves

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] Nonlinear statistical modeling -- a comparison of R and AD Model Builder

dave fournier
In reply to this post by Douglas Bates-2
Douglas Bates wrote:

>
     snip
>
> Don't you find it somewhat disingenuous that you publish a comparison
> between the AD Model Builder software that you sell and R - a
> comparison that shows a tremendous advantage for your software - and
> then you write "I am not proficient in R"?
>
I think there is a misunderstanding here. I did not pick this example.
As I said it was undertaken by Schnute and his colleagues. I had nothing
to do with it except of course to sell them my software. However as I
stated at that time Splus could not run the model without crashing after
a time so that no comparison was possible. I was aware of those results
and decided to use R to complete the comparison. I don't see why Schnute
would want to unfairly promote my software. I believe he was simply
looking for the best tool for the job he had in mind.

> Had you been proficient in R you might have known about the symbolic
> differentiation capabilities, specifically the deriv function, that


> have been part of the S language since the late 1980s.  I believe that
> the 'AD' in "AD model builder" stands for automatic differentiation,
> which is actually something that John Chambers and I discussed at
> length when we were developing nonlinear modeling methods for S.  In
> the end we went with symbolic differentiation rather than automatic
> differentiation because we felt that symbolic was more flexible.
>
Yes I am aware of the symbolic differentiation capabilities.
I have checked out the deriv function and it does not seem to be capable
of calculating derivatives for a model of this complexity in an
efficient manner. Of course I could be wrong.

There is a paper by
Andreas Griewank (whose title I have forgotten but perhaps some list
member recalls) around 1990 where he compares symbolic and automatic
differentiation for a simple model of an oil reservoir. He demonstrates
quite decisively that symbolic differentiation is not the way to go.


> This is not to say that automatic differentiation isn't a perfectly
> legitimate technique.  However, my recollection is that it would have
> required extensive revisions to the arithmetic expression evaluator,
> which is already very tricky code because of the "recycling rule" and
> the desire to shield users from knowledge of the internal
> representations and such details as whether you are using logical or
> integer or double precision operands or a combination.
>
> If you want to see these details you can, of course, examine the
> source code.  I don't believe we would have the opportunity to examine
> how you implemented automatic differentiation.
>
> I must also agree with Spencer Graves that when I start reading a
> description of a nonlinear model with over 100 parameters, the example
> that you chose, I immediately start thinking of nonlinear mixed
> effects models.  In my experience the only way in which a nonlinear
> model ends up with that number of parameters is through applying an
> underlying model with a low number of parameters to various groups
> within the data.  Table 2 in the Schnute et al. paper to which you
> make reference states that the number of parameters in the model is T
> + A + 5 where T is the number of years of data and A is the number of
> age classes.  To me that looks a lot like a nonlinear mixed effects
> model.
>
I agree that this makes a good nonlinear random effects example. Of
course 10 years ago AD Model Builder did not have that capability. It
now does and my colleague Hans Skaug has modified the code to
incorporate random effects. I believe the model converges in a few
minutes. He will report the results and hopefully they can be compared
to nlme or any other software in R which can carry out the calculations.


> Also, your choice of subject heading for your message seems
> deliberatively provocative.  You seem to be implying that you are
> discussing a comparisons of AD Model Builder and R on all aspects of
> nonlinear statistical modeling but you are only discussing one
> comparison on simulated data using a model from the applications area
> for which you wrote AD Model Builder.  Then you follow up by saying "I
> am not proficient in R" and your results for R are from applying code
> that someone else gave you.
>
> It seems that ADMB had a bit of a "home-field advantage" in this
> particular comparison.
I don't quite get your point. Of course I am only going to present
examples where I believe ADMB is (far) superior to R. Otherwise I would
just be wasting everyones time. ADMB is much more narrowly focused than
R. I think that people can examine the arguments and make up their own
minds.

>
> I view nonlinear statistical modeling differently.  I have had a bit
> of experience in the area and I find that I want to examine the data
> carefully, usually through plots, before I embark on fitting
> complicated models.  I like to have some assurance that the model
> makes sense in the context of the data.  (In your example you don't
> need to worry about appropriateness of the model because the data were
> simulated.) I would never try to fit a nonlinear model with 100
> parameters to data without carefully examining the data, and
> especially selected subsets of the data, first.  For this the
> flexibility of the S language and tools like lattice graphics that
> were developed in this language are invaluable to me.  The flexibility
> of data manipulation and graphics for interactive exploration of data
> is what attracted me to S in the first place.
>
> I realize that for many people the area of nonlinear statistical
> modeling is reduced to "Fit this model to these data and don't ask any
> questions.   Just give me parameter estimates and p-values."  If that
> is your situation then it would make sense to use software that gets
> you those estimates as quickly as possible with a minimum of effort.
> I'm just happy that I get to turn down people who ask me to do that.
> I like that fact that I can spend my time asking questions about the
> data and of the data.
>
    snip

--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com



Douglas Bates wrote:

> On 11/24/06, dave fournier <[hidden email]> wrote:
>>
>>
>>        Dave
>>  > Did you try supplying gradient information to nlminb?  (I note that
>> nlminb is used for the optimization, but I don't see any gradient
>> information supplied to it.) I would suspect that supplying gradient
>> information would greatly speed up the computation (as you note in
>> comments at http://otter-rsch.ca/tresults.htm.)
>
>> Actually you should probably ask Norm Olsen these questions.
>> I am not proficient in R and am just using his code.
>
> Don't you find it somewhat disingenuous that you publish a comparison
> between the AD Model Builder software that you sell and R - a
> comparison that shows a tremendous advantage for your software - and
> then you write "I am not proficient in R"?
>
> Had you been proficient in R you might have known about the symbolic
> differentiation capabilities, specifically the deriv function, that
> have been part of the S language since the late 1980s.  I believe that
> the 'AD' in "AD model builder" stands for automatic differentiation,
> which is actually something that John Chambers and I discussed at
> length when we were developing nonlinear modeling methods for S.  In
> the end we went with symbolic differentiation rather than automatic
> differentiation because we felt that symbolic was more flexible.
>
> This is not to say that automatic differentiation isn't a perfectly
> legitimate technique.  However, my recollection is that it would have
> required extensive revisions to the arithmetic expression evaluator,
> which is already very tricky code because of the "recycling rule" and
> the desire to shield users from knowledge of the internal
> representations and such details as whether you are using logical or
> integer or double precision operands or a combination.
>
> If you want to see these details you can, of course, examine the
> source code.  I don't believe we would have the opportunity to examine
> how you implemented automatic differentiation.
>
> I must also agree with Spencer Graves that when I start reading a
> description of a nonlinear model with over 100 parameters, the example
> that you chose, I immediately start thinking of nonlinear mixed
> effects models.  In my experience the only way in which a nonlinear
> model ends up with that number of parameters is through applying an
> underlying model with a low number of parameters to various groups
> within the data.  Table 2 in the Schnute et al. paper to which you
> make reference states that the number of parameters in the model is T
> + A + 5 where T is the number of years of data and A is the number of
> age classes.  To me that looks a lot like a nonlinear mixed effects
> model.
>
> Also, your choice of subject heading for your message seems
> deliberatively provocative.  You seem to be implying that you are
> discussing a comparisons of AD Model Builder and R on all aspects of
> nonlinear statistical modeling but you are only discussing one
> comparison on simulated data using a model from the applications area
> for which you wrote AD Model Builder.  Then you follow up by saying "I
> am not proficient in R" and your results for R are from applying code
> that someone else gave you.
>
> It seems that ADMB had a bit of a "home-field advantage" in this
> particular comparison.
>
> I view nonlinear statistical modeling differently.  I have had a bit
> of experience in the area and I find that I want to examine the data
> carefully, usually through plots, before I embark on fitting
> complicated models.  I like to have some assurance that the model
> makes sense in the context of the data.  (In your example you don't
> need to worry about appropriateness of the model because the data were
> simulated.) I would never try to fit a nonlinear model with 100
> parameters to data without carefully examining the data, and
> especially selected subsets of the data, first.  For this the
> flexibility of the S language and tools like lattice graphics that
> were developed in this language are invaluable to me.  The flexibility
> of data manipulation and graphics for interactive exploration of data
> is what attracted me to S in the first place.
>
> I realize that for many people the area of nonlinear statistical
> modeling is reduced to "Fit this model to these data and don't ask any
> questions.   Just give me parameter estimates and p-values."  If that
> is your situation then it would make sense to use software that gets
> you those estimates as quickly as possible with a minimum of effort.
> I'm just happy that I get to turn down people who ask me to do that.
> I like that fact that I can spend my time asking questions about the
> data and of the data.
>
>
>> However I can say that providing derivatives for such a model is a
>> highly nontrivial exercise. As I said in my posting, the  R script and
>> data are available to anyone who feels that the exercise was not carried
>> out properly and would like to improve on it. Also one does not need
>> to provide derivatives to the AD Model Builder program.
>>
>> Finally suppose that you are very good at calculating derivatives and
>> manage to get them right. Then someone else comes along who wants to
>> modify the model. Unless they are also very good at calculating
>> derivatives there will be trouble.
>>
>>  >
>>  > I'm curious -- when you say "R may not be a suitable platform for
>> development for such models", what aspect of R do you feel is lacking?
>> Is it the specific optimization routines available, or is it some other
>> more general aspect?
>>
>> 2 seconds vs 90 minutes. For a real problem of tihs type the timings
>> would probably be something like 10 minutes vs more than 2,700 minutes.
>>
>>  >
>>  > Also, another optimization algorithm available in R is the "L-BFGS-B"
>> method for optim() in the MASS package.  I've had extremely good
>> experiences with using this code in S-PLUS.  It can take box
>> constraints, and can use gradient information.  It is my first choice
>> for most optimization problems, and I believe it is very widely used.
>> Did you try using that optimization routine with this problem?
>>  >
>>  > -- Tony Plate
>>  >
>>  > dave fournier wrote:
>>  >> There has recently been some discussion on the list about
>>  >> AD Model builder and the suitability of R for constructing the
>>  >> types of models used in fisheries management.
>>  >>
>>  >>    https://stat.ethz.ch/pipermail/r-help/2006-January/086841.html
>>  >>
>>  >>    https://stat.ethz.ch/pipermail/r-help/2006-January/086858.html
>>  >>
>>  >> I  think that many R users understimate the numerical challenges
>>  >> that some of the typical nonlinear statistical model used in
>> different
>>  >> fields present. R may not be a suitable platform for development for
>>  >> such models.
>>  >>
>>  >> Around 10 years ago John Schnute, Laura Richards, and Norm Olsen
>>  >> with Canadian federal fisheries undertook an investigation
>>  >> comparing various statistical modeling packages for a simple
>>  >> age-structured statistical model of the type commonly used in
>>  >> fisheries. They compared AD Mdel Builder, Gauss, Matlab, and
>>  >> Splus. Unfortunately a working model could not be produced with Splus
>>  >> so its times could not be included in the comparison. It is possible
>>  >> to produce a working model with the present day version of R so that
>>  >> R can now be directly compared with AD Model Builder for this type
>> of model.
>>  >>
>>  >> I have put the results of the test together with the original
>>  >> Schnute and Richards paper and the working R and AD Model Builder
>>  >> codes on Otter's web site
>>  >>
>>  >>      http://otter-rsch.ca/tresults.htm
>>  >>
>>  >> The results are that AD Model builder is roughly 1000 times faster
>> than
>>  >> R for this problem. ADMB takes about 2 seconds to converge while
>>  >> R takes over 90 minutes.
>>  >>
>>  >> This is a simple toy example. Real fisheries models are often
>> hundred of
>>  >> times more computationally intensive as this one.
>>  >>
>>  >>         Cheers,
>>  >>
>>  >>          Dave
>>  >> ~
>>  >
>>
>>
>> --
>> David A. Fournier
>> P.O. Box 2040,
>> Sidney, B.C. V8l 3S3
>> Canada
>> Phone/FAX 250-655-3364
>> http://otter-rsch.com
>>
>> --
>> David A. Fournier
>> P.O. Box 2040,
>> Sidney, B.C. V8l 3S3
>> Canada
>> Phone/FAX 250-655-3364
>> http://otter-rsch.com
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>


--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.