[R] lme4 package: Fitted values and residuals

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[R] lme4 package: Fitted values and residuals

Frank Johannes
Dear all,
I have three concerns:
1)
I am running models with the lme4 package. I cannot find a way to pull
out a vector of the fitted values and the residuals. Does anybody know
how to do it?
2)
How can I nest a random effect variable into a "two-level" fixed effect
variable?
3)
Suppose I have the following model:
y = a + b|c + d + error,
where 'a' is a fixed effect, 'c' is a random effect nested with the
random effect 'b', and 'd' is  a non-nested random effect.
Suppose I obtain parameter estimates for all of the predictor variables.
 Now, I would like to calculate a fitted value, 'yhat', from only the
parameter estimates for 'a' and 'b|c'. Can this be done in lme4. That
is; can I pull out all of the requisite elements from the output to
calculate 'yhat'?
Thanks for your help,
Frank.


--

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] lme4 package: Fitted values and residuals

Douglas Bates-2
On 11/16/06, Frank Johannes <[hidden email]> wrote:
> Dear all,
> I have three concerns:
> 1)
> I am running models with the lme4 package. I cannot find a way to pull
> out a vector of the fitted values and the residuals. Does anybody know
> how to do it?

The fitted() and resid() extractor functions are the usual way of
doing this for a fitted model in R and, astonishingly enough, they
work!  Try

library(lme4)
example(lmer)
fitted(fm1)
resid(fm1)

> 2)
> How can I nest a random effect variable into a "two-level" fixed effect
> variable?

I'm not quite sure what you mean.  Are you talking about a situation
like having a random effect for "Patient" and a fixed effect for
"Treatment" with Patient nested within Treatment.  If the Patients are
designated in a sensible way, so that each Patient has a distinct
identifier (i.e. the number of levels of the Patient factor
corresponds to the number of Patients) then you can ignore the nesting
and specify the model as

lmer(response ~ Treatment + (1|Patient), ...)

Sometimes the Patients are designated with implicit nesting so that
the treatment group has a patient labeled "1" and the control group
has another patient also labeled as "1".  In that case you need to
specify the grouping factor for the random effects as the
Treatment:Patient interaction.  That is,

lmer(response ~ Treatment + (1|Treatment:Patient), ...)

The second version also works properly if the Patients had been
designated in a sensible way so, if you want to be safe, use the
second version.

> 3)
> Suppose I have the following model:
> y = a + b|c + d + error,
> where 'a' is a fixed effect, 'c' is a random effect nested with the
> random effect 'b', and 'd' is  a non-nested random effect.

That's not the greatest notation to use for the R-Help list because it
doesn't correspond to the way that R formulas are parsed.

> Suppose I obtain parameter estimates for all of the predictor variables.
>  Now, I would like to calculate a fitted value, 'yhat', from only the
> parameter estimates for 'a' and 'b|c'. Can this be done in lme4. That
> is; can I pull out all of the requisite elements from the output to
> calculate 'yhat'?

The answer to the question of "Can this be done in lme4?" is

install.packages("fortunes"); library(fortunes); fortune("Yoda")

I could describe how to do it if I had a better idea of what the
calculation you had in mind was.  I am having difficulty grasping how
you obtain a prediction from a model involving a factor 'd' without
assigning a value to that factor.  I suppose that someone who
understands the deep magic of quantities like SAS's LS-means may be
able to tell you what you want to know but I can't.

I can say that you can obtain the coefficient estimates for the fixed
effects with

fixef(fm1)

and a list of the conditional modes of the random effects (also called
the BLUPs) as

ranef(fm1)

After that you are on your own.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] lme4 package: Fitted values and residuals

Renaud Lancelot
2006/11/16, Douglas Bates <[hidden email]>:

> On 11/16/06, Frank Johannes <[hidden email]> wrote:
> > Dear all,
> > I have three concerns:
> > 1)
> > I am running models with the lme4 package. I cannot find a way to pull
> > out a vector of the fitted values and the residuals. Does anybody know
> > how to do it?
>
> The fitted() and resid() extractor functions are the usual way of
> doing this for a fitted model in R and, astonishingly enough, they
> work!  Try
>
> library(lme4)
> example(lmer)
> fitted(fm1)
> resid(fm1)

But resid does not seem to work with GLMMs:

> library(lme4)   ##  version 0.9975-9
Le chargement a nécessité le package : Matrix
Le chargement a nécessité le package : lattice
> m1 <- lmer(cbind(incidence, size - incidence) ~ period + (1 | herd),
+             family = binomial, data = cbpp)
relative tolerance set to 9.8721686386158e-05
> resid(m1)
Erreur : 'resid' n'est pas encore implémenté

[in English: Error: 'resid' is not implemented yet]

Best,

Renaud

>
> > 2)
> > How can I nest a random effect variable into a "two-level" fixed effect
> > variable?
>
> I'm not quite sure what you mean.  Are you talking about a situation
> like having a random effect for "Patient" and a fixed effect for
> "Treatment" with Patient nested within Treatment.  If the Patients are
> designated in a sensible way, so that each Patient has a distinct
> identifier (i.e. the number of levels of the Patient factor
> corresponds to the number of Patients) then you can ignore the nesting
> and specify the model as
>
> lmer(response ~ Treatment + (1|Patient), ...)
>
> Sometimes the Patients are designated with implicit nesting so that
> the treatment group has a patient labeled "1" and the control group
> has another patient also labeled as "1".  In that case you need to
> specify the grouping factor for the random effects as the
> Treatment:Patient interaction.  That is,
>
> lmer(response ~ Treatment + (1|Treatment:Patient), ...)
>
> The second version also works properly if the Patients had been
> designated in a sensible way so, if you want to be safe, use the
> second version.
>
> > 3)
> > Suppose I have the following model:
> > y = a + b|c + d + error,
> > where 'a' is a fixed effect, 'c' is a random effect nested with the
> > random effect 'b', and 'd' is  a non-nested random effect.
>
> That's not the greatest notation to use for the R-Help list because it
> doesn't correspond to the way that R formulas are parsed.
>
> > Suppose I obtain parameter estimates for all of the predictor variables.
> >  Now, I would like to calculate a fitted value, 'yhat', from only the
> > parameter estimates for 'a' and 'b|c'. Can this be done in lme4. That
> > is; can I pull out all of the requisite elements from the output to
> > calculate 'yhat'?
>
> The answer to the question of "Can this be done in lme4?" is
>
> install.packages("fortunes"); library(fortunes); fortune("Yoda")
>
> I could describe how to do it if I had a better idea of what the
> calculation you had in mind was.  I am having difficulty grasping how
> you obtain a prediction from a model involving a factor 'd' without
> assigning a value to that factor.  I suppose that someone who
> understands the deep magic of quantities like SAS's LS-means may be
> able to tell you what you want to know but I can't.
>
> I can say that you can obtain the coefficient estimates for the fixed
> effects with
>
> fixef(fm1)
>
> and a list of the conditional modes of the random effects (also called
> the BLUPs) as
>
> ranef(fm1)
>
> After that you are on your own.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


--
Renaud LANCELOT
Département Elevage et Médecine Vétérinaire (EMVT) du CIRAD
Directeur adjoint chargé des affaires scientifiques

CIRAD, Animal Production and Veterinary Medicine Department
Deputy director for scientific affairs

Campus international de Baillarguet
TA 30 / B (Bât. B, Bur. 214)
34398 Montpellier Cedex 5 - France
Tél   +33 (0)4 67 59 37 17
Secr. +33 (0)4 67 59 39 04
Fax   +33 (0)4 67 59 37 95

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] lme4 package: Fitted values and residuals

Joel Dubin
In reply to this post by Frank Johannes
Hello,

Indeed, implementing the fitted and resid calls on the fm1 object from example(lmer), the fitted function worked, but the resid did not:

> resid(fm1)
Error in resid(fm1) : no slot of name "family" for this object of class "lmer"

I am using R 2.4.0 (on Windows XP), with the update of lme4 (lmer within) written for 2.4.0.

Thanks, Joel.




> Message: 79 Date: Fri, 17 Nov 2006 03:14:24 +0100 From: "Renaud
> Lancelot" <[hidden email]> Subject: Re: [R] lme4 package:
> Fitted values and residuals To: "Douglas Bates" <[hidden email]>
> Cc: [hidden email], Frank Johannes <[hidden email]>
> Message-ID:
> <[hidden email]>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 2006/11/16, Douglas Bates <[hidden email]>:
>> > On 11/16/06, Frank Johannes <[hidden email]> wrote:
>>> > > Dear all,
>>> > > I have three concerns:
>>> > > 1)
>>> > > I am running models with the lme4 package. I cannot find a way to pull
>>> > > out a vector of the fitted values and the residuals. Does anybody know
>>> > > how to do it?
>> >
>> > The fitted() and resid() extractor functions are the usual way of
>> > doing this for a fitted model in R and, astonishingly enough, they
>> > work!  Try
>> >
>> > library(lme4)
>> > example(lmer)
>> > fitted(fm1)
>> > resid(fm1)
>
> But resid does not seem to work with GLMMs:
>
>> > library(lme4)   ##  version 0.9975-9
> Le chargement a n?cessit? le package : Matrix
> Le chargement a n?cessit? le package : lattice
>> > m1 <- lmer(cbind(incidence, size - incidence) ~ period + (1 | herd),
> +             family = binomial, data = cbpp)
> relative tolerance set to 9.8721686386158e-05
>> > resid(m1)
> Erreur : 'resid' n'est pas encore impl?ment?
>
> [in English: Error: 'resid' is not implemented yet]
>
> Best,
>
> Renaud
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] lme4 package: Fitted values and residuals

Martin Maechler
>>>>> "Joel" == Joel Dubin <[hidden email]>
>>>>>     on Fri, 17 Nov 2006 16:19:16 -0500 writes:

    Joel> Hello, Indeed, implementing the fitted and resid calls
    Joel> on the fm1 object from example(lmer), the fitted
    Joel> function worked, but the resid did not:

    >> resid(fm1)
    Joel> Error in resid(fm1) : no slot of name "family" for
    Joel> this object of class "lmer"

    Joel> I am using R 2.4.0 (on Windows XP), with the update of
    Joel> lme4 (lmer within) written for 2.4.0.

I think you should update lme4 [ update.packages(.....) ],
since the above looks suspiciously like a bug that had been
fixed not so long ago.

IIRC, the posting guide asks you to provide  sessionInfo()
in such situations.

Regards,
Martin

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] lme4 package: Fitted values and residuals

Joel Dubin
Thank you, Martin, for pointing this out (not to mention the missing
sessionInfo() piece).  Indeed the resid() call works as expected with
the most recent update of lme4.

Cheers, Joel.


Martin Maechler wrote:

>>>>>> "Joel" == Joel Dubin <[hidden email]>
>>>>>>     on Fri, 17 Nov 2006 16:19:16 -0500 writes:
>>>>>>            
>
>     Joel> Hello, Indeed, implementing the fitted and resid calls
>     Joel> on the fm1 object from example(lmer), the fitted
>     Joel> function worked, but the resid did not:
>
>     >> resid(fm1)
>     Joel> Error in resid(fm1) : no slot of name "family" for
>     Joel> this object of class "lmer"
>
>     Joel> I am using R 2.4.0 (on Windows XP), with the update of
>     Joel> lme4 (lmer within) written for 2.4.0.
>
> I think you should update lme4 [ update.packages(.....) ],
> since the above looks suspiciously like a bug that had been
> fixed not so long ago.
>
> IIRC, the posting guide asks you to provide  sessionInfo()
> in such situations.
>
> Regards,
> Martin
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] lme4 package: Fitted values and residuals

Renaud Lancelot
Not quite:

> library(lme4)
> sessionInfo()
R version 2.4.0 (2006-10-03)
i386-pc-mingw32

locale:
LC_COLLATE=French_France.1252;LC_CTYPE=French_France.1252;LC_MONETARY=French_France.1252;LC_NUMERIC=C;LC_TIME=French_France.1252

attached base packages:
[1] "methods"   "stats"     "graphics"  "grDevices" "utils"
"datasets"  "tcltk"     "base"

other attached packages:
      lme4     Matrix    lattice   fortunes       svIO     R2HTML
svMisc   svSocket      svIDE
"0.9975-9" "0.9975-6"  "0.14-13"    "1.3-2"    "0.9-5"     "1.58"
"0.9-5"    "0.9-5"    "0.9-5"
> example(cbpp)

cbpp> (m1 <- lmer(cbind(incidence, size - incidence) ~ period +
    (1 | herd), family = binomial, data = cbpp))
[snip]
> head(fitted(m1))
[1] -0.803955 -1.807372 -1.945307 -2.388617 -1.692780 -2.696197
> head(resid(m1))
Erreur : 'resid' n'est pas encore implémenté
[snip]

Best,

Renaud




2006/11/18, Joel Dubin <[hidden email]>:

> Thank you, Martin, for pointing this out (not to mention the missing
> sessionInfo() piece).  Indeed the resid() call works as expected with
> the most recent update of lme4.
>
> Cheers, Joel.
>
>
> Martin Maechler wrote:
> >>>>>> "Joel" == Joel Dubin <[hidden email]>
> >>>>>>     on Fri, 17 Nov 2006 16:19:16 -0500 writes:
> >>>>>>
> >
> >     Joel> Hello, Indeed, implementing the fitted and resid calls
> >     Joel> on the fm1 object from example(lmer), the fitted
> >     Joel> function worked, but the resid did not:
> >
> >     >> resid(fm1)
> >     Joel> Error in resid(fm1) : no slot of name "family" for
> >     Joel> this object of class "lmer"
> >
> >     Joel> I am using R 2.4.0 (on Windows XP), with the update of
> >     Joel> lme4 (lmer within) written for 2.4.0.
> >
> > I think you should update lme4 [ update.packages(.....) ],
> > since the above looks suspiciously like a bug that had been
> > fixed not so long ago.
> >
> > IIRC, the posting guide asks you to provide  sessionInfo()
> > in such situations.
> >
> > Regards,
> > Martin
> >
>


--
Renaud LANCELOT
Département Elevage et Médecine Vétérinaire (EMVT) du CIRAD
Directeur adjoint chargé des affaires scientifiques

CIRAD, Animal Production and Veterinary Medicine Department
Deputy director for scientific affairs

Campus international de Baillarguet
TA 30 / B (Bât. B, Bur. 214)
34398 Montpellier Cedex 5 - France
Tél   +33 (0)4 67 59 37 17
Secr. +33 (0)4 67 59 39 04
Fax   +33 (0)4 67 59 37 95

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.