nlme: problem with fitting logistic function

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

nlme: problem with fitting logistic function

jakub kreisinger
Dear colegues
I am trying to analyze growth data on mice. To do this I attempted to fit logistic curve using nlme package. However, the dataset I use is large (in total ca 20 000 measures on ca 3 000 individuals) with relatively complicated structure (several explanatory variables with interactions + random effect where individual offspring are nested within particular litters are nested within particular parental pairs).
Although I had no problems to fit models, where the complexity of random effect was reduce (which is conceptually incorrect), the fitting procedure of the full model did not reach the 1 iteration after several hours. Do you have any idea how to solve this problem?

Thank you very much for your advice
Best regards
Jakub Kreisinger

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: nlme: problem with fitting logistic function

Dieter Menne
jakub kreisinger <jakubkreisinger <at> seznam.cz> writes:

> I am trying to analyze growth data on mice. To do this I attempted to fit
logistic curve using nlme package.
> However, the dataset I use is large (in total ca 20 000 measures on ca 3 000
individuals) with relatively
> complicated structure (several explanatory variables with interactions +
random effect where
> individual offspring are nested within particular litters are nested within
particular parental
> pairs).
> Although I had no problems to fit models, where the complexity of random
effect was reduce (which is
> conceptually incorrect), the fitting procedure of the full model did not reach
the 1 iteration after
> several hours. Do you have any idea how to solve this problem?

1) Get better starting values by using nlsList
2) Get still better starting values by fitting a simpler model first with
   nlme. This is VERY successful for me, it made many problems feasibly that
   blew up otherwise. Good luck in finding the right syntax for complex
   start values, this can be a huge challenge.
3) Use lmer in lme4. Your mileage may vary, I could not find a speedup
   for my problems, but larger problem might give one.
4) Use C for the core function. This is very effective, and there is at least
   on example coming with nlme (was it SSlogist?).

Dieter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: nlme: problem with fitting logistic function

Dieter Menne
Dieter Menne <dieter.menne <at> menne-biomed.de> writes:

> 4) Use C for the core function. This is very effective, and there is at least
>    on example coming with nlme (was it SSlogist?).

I correct: it is phenoModel, calling nlme_one_comp_first in nlme.c

Dieter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: nlme: problem with fitting logistic function

Douglas Bates-2
In reply to this post by Dieter Menne
On Mon, Mar 9, 2009 at 12:13 PM, Dieter Menne
<[hidden email]> wrote:

> jakub kreisinger <jakubkreisinger <at> seznam.cz> writes:
>
>> I am trying to analyze growth data on mice. To do this I attempted to fit
> logistic curve using nlme package.
>> However, the dataset I use is large (in total ca 20 000 measures on ca 3 000
> individuals) with relatively
>> complicated structure (several explanatory variables with interactions +
> random effect where
>> individual offspring are nested within particular litters are nested within
> particular parental
>> pairs).
>> Although I had no problems to fit models, where the complexity of random
> effect was reduce (which is
>> conceptually incorrect), the fitting procedure of the full model did not reach
> the 1 iteration after
>> several hours. Do you have any idea how to solve this problem?
>
> 1) Get better starting values by using nlsList
> 2) Get still better starting values by fitting a simpler model first with
>   nlme. This is VERY successful for me, it made many problems feasibly that
>   blew up otherwise. Good luck in finding the right syntax for complex
>   start values, this can be a huge challenge.
> 3) Use lmer in lme4. Your mileage may vary, I could not find a speedup
>   for my problems, but larger problem might give one.

Did you mean nlmer in the lme4 package?  If so, it may be worthwhile
trying the development branch but that is not something for the
faint-hearted.

> 4) Use C for the core function. This is very effective, and there is at least
>   on example coming with nlme (was it SSlogist?).

Do you think that evaluation of the model function takes a substantial
portion of the computing time?  I am asking for my interest, not
because I think I know the answer.  So, for example, have you profiled
difficult nlme fits and found that the model function evaluation was
expensive?

I did find a similar result when fitting generalized linear mixed
models - a substantial portion of the execution time was spent in the
evaluation of the inverse link function and its derivative - so I
moved that to compiled code.  Interestingly in those cases it wasn't
the evaluation of the function as much as checking the boundary
conditions that was taking up time.

I wouldn't recommend patterning such code after nlme_one_comp_first in
nlme.c  I would use the .Call interface to C code instead.  The real
trick would be applying the recycling rule inside the C code without
tying yourself in knots.  A function like SSlogis (there is no "t" in
the name, by the way) takes an "input" argument and three parameters,
"Asym", "xmid" and "scal".   The "input" argument always has a length
which is the number of observations but "Asym", "xmid" and "scal" are
sometimes scalars and sometimes vectors of the same length as "input".

If you would be willing to run some tests, I will volunteer to write
the code and we can see if it helps.  (Not to say that you couldn't
write it yourself if you were so inclined but I do this a lot and
could probably get something working relatively quickly - why do
those sound like "famous last words"?)

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: nlme: problem with fitting logistic function

Dieter Menne
Douglas Bates <bates <at> stat.wisc.edu> writes:

> > 3) Use lmer in lme4. Your mileage may vary, I could not find a speedup
> >   for my problems, but larger problem might give one.
>
> Did you mean nlmer in the lme4 package?  If so, it may be worthwhile
> trying the development branch but that is not something for the
> faint-hearted.

Thanks, Doug, for you comments. To be fair, I wrote these unordered
thoughts to get you out of the snowhole :-)
 
> > 4) Use C for the core function. This is very effective, and there is at least
> >   on example coming with nlme (was it SSlogist?).
>
> Do you think that evaluation of the model function takes a substantial
> portion of the computing time?  I am asking for my interest, not
> because I think I know the answer.  So, for example, have you profiled
> difficult nlme fits and found that the model function evaluation was
> expensive?

No, I did not profile that function, but 8 years ago tried it once because
at that time I thought it would help. Nowadays, I am more inclined to think
that failure of nlme with a more complex model is a failure of the model,
not of nlme; I don't have speed problems with my data.

However, I remember that in a similar case with ode/lsoda, using c function
made a factor of 20++, so I would have a look into it again if speed
was a concern for me.

Dieter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: nlme: problem with fitting logistic function

Dieter Menne
In reply to this post by Douglas Bates-2
Douglas Bates <bates <at> stat.wisc.edu> writes:

> > 3) Use lmer in lme4. Your mileage may vary, I could not find a speedup
> >   for my problems, but larger problem might give one.
>
> Did you mean nlmer in the lme4 package?  If so, it may be worthwhile
> trying the development branch but that is not something for the
> faint-hearted.

Thanks, Doug, for you comments. To be fair, I wrote these unordered
thoughts to get you out of the snowhole :-)
 
> > 4) Use C for the core function. This is very effective, and there is at least
> >   on example coming with nlme (was it SSlogist?).
>
> Do you think that evaluation of the model function takes a substantial
> portion of the computing time?  I am asking for my interest, not
> because I think I know the answer.  So, for example, have you profiled
> difficult nlme fits and found that the model function evaluation was
> expensive?

No, I did not profile that function, but 8 years ago tried it once because
at that time I thought it would help. Nowadays, I am more inclined to think
that failure of nlme with a more complex model is a failure of the model,
not of nlme; I don't have speed problems with my data.

However, I remember that in a similar case with ode/lsoda, using c function
made a factor of 20++, so I would have a look into it again if speed
was a concern for me.

Dieter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: nlme: problem with fitting logistic function

dave fournier
In reply to this post by jakub kreisinger
I think you can do this very efficiently with AD Model Builder's
random effects module. The software is now freely available at

   http://admb-project.org

If you want, you can contact me directly to discuss the model.

    Dave

--
David A. Fournier
P.O. Box 2040,
Sidney, B.C. V8l 3S3
Canada
Phone/FAX 250-655-3364
http://otter-rsch.com

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.