nonlinear model: pseudo-design matrix

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

nonlinear model: pseudo-design matrix

Murray Jorgensen
Given a nonlinear model formula and a set of values for all the
parameters defining a point in parameter space, is there a neat way to
extract the pseudodesign matrix of the model at the point? That is the
matrix of partial derivatives of the fitted values w.r.t. the parameters
evaluated at the point.

(I have figured out how to extract the gradient information from an nls
fitted model using the nlsModel part, but I wish to implement a score
test, so I need to be able to extract the information at points other
than the mle.)

Thanks, Murray Jorgensen
--
Dr Murray Jorgensen      http://www.stats.waikato.ac.nz/Staff/maj.html
Department of Statistics, University of Waikato, Hamilton, New Zealand
Email: [hidden email]                                Fax 7 838 4155
Phone  +64 7 838 4773 wk    Home +64 7 825 0441    Mobile 021 1395 862

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: nonlinear model: pseudo-design matrix

Spencer Graves
          There doubtless is a way to extract the gradient information you
desire, but have you considered profiling instead?  Are you familiar
with the distinction between intrinsic and parameter effects curvature?
  In brief, part of the nonlinearities involved in nonlinear least
squares are intrinsic to the problem, and part are due to the how the
problem is parameterized.  If you change the parameterization, you
change the parameter effects curvature, but the intrinsic curvature
remains unchanged.  Roughly 30 years ago, Doug Bates and Don Watts
reanalized a few dozen published nonlinear regression fits, and found
that in all but perhaps one or two, the parameter effects were dominant
and the intrinsic curvature was negligible.  See Bates and Watts (1988)
Nonlinear Regression Analysis and Its Applications (Wiley) or Seber and
Wild (1989) Nonlinear Regression (Wiley).

          Bottom line:

          1.  You will always get more accurate answers from profiling than
from the Wald "pseudodesign matrix" approach.  Moreover, often the
differences are dramatic.

          2.  I just did RSiteSearch("profiling with nls").  The first hit was
"http://finzi.psych.upenn.edu/R/library/stats/html/profile.nls.html".
If this is not satisfactory, please explain why.

          hope this helps.
          spencer graves

Murray Jorgensen wrote:

> Given a nonlinear model formula and a set of values for all the
> parameters defining a point in parameter space, is there a neat way to
> extract the pseudodesign matrix of the model at the point? That is the
> matrix of partial derivatives of the fitted values w.r.t. the parameters
> evaluated at the point.
>
> (I have figured out how to extract the gradient information from an nls
> fitted model using the nlsModel part, but I wish to implement a score
> test, so I need to be able to extract the information at points other
> than the mle.)
>
> Thanks, Murray Jorgensen

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: nonlinear model: pseudo-design matrix

Murray Jorgensen
Hi Spencer,

you were the only one to reply. Yes I am aware of the intrinsic /
parameter effects distinction and the advantages of LR tests and
profiling over Wald tests based on the local curvature of the
loglikelihood surface at the larger of two models being compared. My
situation is that I am comparing two nested models both of which have
uncomfortably many parameters for the amount of data available. I am
able to fit the smaller of the two models but not the larger. In this
situation neither the the Wald nor the LR test is available to me but
the score test (a.k.a. the Lagrange Multiplier test) is available to me
because it is based on the loglikelihood gradient at the smaller model.

I have been able to carry out the test by extracting

X <- smaller.nls$m$gradient()

and obtaining the extra columns of X for the parameters in larger but
not in smaller by numerical differentiation. It seems that there should
be some way of obtaining the extra columns without recourse to numerical
differentiation, though.

Cheers,  Murray Jorgensen

Spencer Graves wrote:

>       There doubtless is a way to extract the gradient information you
> desire, but have you considered profiling instead?  Are you familiar
> with the distinction between intrinsic and parameter effects curvature?
>  In brief, part of the nonlinearities involved in nonlinear least
> squares are intrinsic to the problem, and part are due to the how the
> problem is parameterized.  If you change the parameterization, you
> change the parameter effects curvature, but the intrinsic curvature
> remains unchanged.  Roughly 30 years ago, Doug Bates and Don Watts
> reanalized a few dozen published nonlinear regression fits, and found
> that in all but perhaps one or two, the parameter effects were dominant
> and the intrinsic curvature was negligible.  See Bates and Watts (1988)
> Nonlinear Regression Analysis and Its Applications (Wiley) or Seber and
> Wild (1989) Nonlinear Regression (Wiley).
>
>       Bottom line:
>
>       1.  You will always get more accurate answers from profiling than
> from the Wald "pseudodesign matrix" approach.  Moreover, often the
> differences are dramatic.
>
>       2.  I just did RSiteSearch("profiling with nls").  The first hit
> was
> "http://finzi.psych.upenn.edu/R/library/stats/html/profile.nls.html". If
> this is not satisfactory, please explain why.
>
>       hope this helps.
>       spencer graves
>
> Murray Jorgensen wrote:
>> Given a nonlinear model formula and a set of values for all the
>> parameters defining a point in parameter space, is there a neat way to
>> extract the pseudodesign matrix of the model at the point? That is the
>> matrix of partial derivatives of the fitted values w.r.t. the parameters
>> evaluated at the point.
>>
>> (I have figured out how to extract the gradient information from an
>> nls fitted model using the nlsModel part, but I wish to implement a
>> score test, so I need to be able to extract the information at points
>> other than the mle.)
>>
>> Thanks, Murray Jorgensen

--
Dr Murray Jorgensen      http://www.stats.waikato.ac.nz/Staff/maj.html
Department of Statistics, University of Waikato, Hamilton, New Zealand
Email: [hidden email]                                Fax 7 838 4155
Phone  +64 7 838 4773 wk    Home +64 7 825 0441    Mobile 021 1395 862

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: nonlinear model: pseudo-design matrix

Spencer Graves
Hi, Murray:

          When I have problems with nonconvergence of nls, I often
move the problem to "optim(..., hessian=TRUE)".  Even if the larger
model is overparameterized and the hessian is singular, I optim usually
returns an answer from which I can then compute 2*log(likelihood ratio).
  Moreover, the hessian will help me diagnose the problem.  If it were
my problem today, I'd try the following:

          (1) If there are substantial differences in the diagonal elements of
the hessian, it suggests the scaling should be adjusted.  Not too long
ago, someone else suggested that this could be done within optim via the
argument control = list(parscale=...).  I have yet to try that, but I
think it should work fine.

          (2) If the diagonal elements of the hessian do not differ by more
than a couple orders of magnitude, then I'd try eigen(fit$hessian,
symmetric=TRUE).  The relative magnitudes of the eigenvalues will expose
the effective numer of paramaters that can be estimated, and the
eigenvectors associated with the smallest eigenvalues can help one
diagnose the problem.

          hope this helps.
          spencer graves

Murray Jorgensen wrote:

> Hi Spencer,
>
> you were the only one to reply. Yes I am aware of the intrinsic /
> parameter effects distinction and the advantages of LR tests and
> profiling over Wald tests based on the local curvature of the
> loglikelihood surface at the larger of two models being compared. My
> situation is that I am comparing two nested models both of which have
> uncomfortably many parameters for the amount of data available. I am
> able to fit the smaller of the two models but not the larger. In this
> situation neither the the Wald nor the LR test is available to me but
> the score test (a.k.a. the Lagrange Multiplier test) is available to me
> because it is based on the loglikelihood gradient at the smaller model.
>
> I have been able to carry out the test by extracting
>
> X <- smaller.nls$m$gradient()
>
> and obtaining the extra columns of X for the parameters in larger but
> not in smaller by numerical differentiation. It seems that there should
> be some way of obtaining the extra columns without recourse to numerical
> differentiation, though.
>
> Cheers,  Murray Jorgensen
>
> Spencer Graves wrote:
>
>>       There doubtless is a way to extract the gradient information you
>> desire, but have you considered profiling instead?  Are you familiar
>> with the distinction between intrinsic and parameter effects
>> curvature?  In brief, part of the nonlinearities involved in nonlinear
>> least squares are intrinsic to the problem, and part are due to the
>> how the problem is parameterized.  If you change the parameterization,
>> you change the parameter effects curvature, but the intrinsic
>> curvature remains unchanged.  Roughly 30 years ago, Doug Bates and Don
>> Watts reanalized a few dozen published nonlinear regression fits, and
>> found that in all but perhaps one or two, the parameter effects were
>> dominant and the intrinsic curvature was negligible.  See Bates and
>> Watts (1988) Nonlinear Regression Analysis and Its Applications
>> (Wiley) or Seber and Wild (1989) Nonlinear Regression (Wiley).
>>
>>       Bottom line:
>>
>>       1.  You will always get more accurate answers from profiling
>> than from the Wald "pseudodesign matrix" approach.  Moreover, often
>> the differences are dramatic.
>>
>>       2.  I just did RSiteSearch("profiling with nls").  The first hit
>> was
>> "http://finzi.psych.upenn.edu/R/library/stats/html/profile.nls.html".
>> If this is not satisfactory, please explain why.
>>
>>       hope this helps.
>>       spencer graves
>>
>> Murray Jorgensen wrote:
>>
>>> Given a nonlinear model formula and a set of values for all the
>>> parameters defining a point in parameter space, is there a neat way to
>>> extract the pseudodesign matrix of the model at the point? That is the
>>> matrix of partial derivatives of the fitted values w.r.t. the parameters
>>> evaluated at the point.
>>>
>>> (I have figured out how to extract the gradient information from an
>>> nls fitted model using the nlsModel part, but I wish to implement a
>>> score test, so I need to be able to extract the information at points
>>> other than the mle.)
>>>
>>> Thanks, Murray Jorgensen
>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: nonlinear model: pseudo-design matrix

Spencer Graves
Hi, Murray:

          I just got 54 hits from RSiteSearch("numerical differentiation"), the
first of which mentioned a function "numericDeriv" WITH a warning
(http://finzi.psych.upenn.edu/R/Rhelp02a/archive/55462.html).

          hope this helps.
          spencer graves
###################
Hi Spencer,

I will try some of the ways you suggest and thank you for the
suggestions. I still think that performing a score test is a sensible
thing to do in my situation, though.

Murray

Spencer Graves wrote:

> Hi, Murray:
>
>       When I have problems with nonconvergence of nls, I often
> move the problem to "optim(..., hessian=TRUE)".  Even if the larger
> model is overparameterized and the hessian is singular, I optim usually
> returns an answer from which I can then compute 2*log(likelihood ratio).
>  Moreover, the hessian will help me diagnose the problem.  If it were my
> problem today, I'd try the following:
>
>       (1) If there are substantial differences in the diagonal elements
> of the hessian, it suggests the scaling should be adjusted.  Not too
> long ago, someone else suggested that this could be done within optim
> via the argument control = list(parscale=...).  I have yet to try that,
> but I think it should work fine.
>
>       (2) If the diagonal elements of the hessian do not differ by more
> than a couple orders of magnitude, then I'd try eigen(fit$hessian,
> symmetric=TRUE).  The relative magnitudes of the eigenvalues will expose
> the effective numer of paramaters that can be estimated, and the
> eigenvectors associated with the smallest eigenvalues can help one
> diagnose the problem.
>
>       hope this helps.
>       spencer graves
>
> Murray Jorgensen wrote:
>
>> Hi Spencer,
>>
>> you were the only one to reply. Yes I am aware of the intrinsic /
>> parameter effects distinction and the advantages of LR tests and
>> profiling over Wald tests based on the local curvature of the
>> loglikelihood surface at the larger of two models being compared. My
>> situation is that I am comparing two nested models both of which have
>> uncomfortably many parameters for the amount of data available. I am
>> able to fit the smaller of the two models but not the larger. In this
>> situation neither the the Wald nor the LR test is available to me but
>> the score test (a.k.a. the Lagrange Multiplier test) is available to
>> me because it is based on the loglikelihood gradient at the smaller
>> model.
>>
>> I have been able to carry out the test by extracting
>>
>> X <- smaller.nls$m$gradient()
>>
>> and obtaining the extra columns of X for the parameters in larger but
>> not in smaller by numerical differentiation. It seems that there
>> should be some way of obtaining the extra columns without recourse to
>> numerical differentiation, though.
>>
>> Cheers,  Murray Jorgensen
>>
>> Spencer Graves wrote:
>>
>>>       There doubtless is a way to extract the gradient information
>>> you desire, but have you considered profiling instead?  Are you
>>> familiar with the distinction between intrinsic and parameter effects
>>> curvature?  In brief, part of the nonlinearities involved in
>>> nonlinear least squares are intrinsic to the problem, and part are
>>> due to the how the problem is parameterized.  If you change the
>>> parameterization, you change the parameter effects curvature, but the
>>> intrinsic curvature remains unchanged.  Roughly 30 years ago, Doug
>>> Bates and Don Watts reanalized a few dozen published nonlinear
>>> regression fits, and found that in all but perhaps one or two, the
>>> parameter effects were dominant and the intrinsic curvature was
>>> negligible.  See Bates and Watts (1988) Nonlinear Regression Analysis
>>> and Its Applications (Wiley) or Seber and Wild (1989) Nonlinear
>>> Regression (Wiley).
>>>
>>>       Bottom line:
>>>
>>>       1.  You will always get more accurate answers from profiling
>>> than from the Wald "pseudodesign matrix" approach.  Moreover, often
>>> the differences are dramatic.
>>>
>>>       2.  I just did RSiteSearch("profiling with nls").  The first
>>> hit was
>>> "http://finzi.psych.upenn.edu/R/library/stats/html/profile.nls.html".
>>> If this is not satisfactory, please explain why.
>>>
>>>       hope this helps.
>>>       spencer graves
>>>
>>> Murray Jorgensen wrote:
>>>
>>>> Given a nonlinear model formula and a set of values for all the
>>>> parameters defining a point in parameter space, is there a neat way to
>>>> extract the pseudodesign matrix of the model at the point? That is the
>>>> matrix of partial derivatives of the fitted values w.r.t. the
>>>> parameters
>>>> evaluated at the point.
>>>>
>>>> (I have figured out how to extract the gradient information from an
>>>> nls fitted model using the nlsModel part, but I wish to implement a
>>>> score test, so I need to be able to extract the information at
>>>> points other than the mle.)
>>>>
>>>> Thanks, Murray Jorgensen
>>
>>
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html