"infinite recursion" in do.call when lme4 loaded only

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

"infinite recursion" in do.call when lme4 loaded only

Dieter Menne
A larg program which worked with lme4/R about a year ago failed when I
re-run it today. I reproduced the problem with the program below.

-- When lme4 is not loaded, the program runs ok and fast enough
-- When lme4 is loaded (but never used), the do.call fails
   with infinite recursion after 60 seconds. Memory used increases
   beyond bonds in task manager.
-- I tested a few S3 based packages (MASS, nlme) and did not get
   similar problems

Current workaround: do lme4-processing in a separate program.


------
#library(lme4) # uncomment this to see the problem
np <- 12
nq <- 20
nph <- 3
nrep <- 30
grd <- expand.grid(Pat=as.factor(1:np),
            Q=as.factor(1:nq),
            Phase=as.factor(1:nph))
df <- with (grd,
  data.frame(Pat=Pat,Q=Q,Phase=Phase,Resp = rnorm(np*nq*nph*nrep)))

score <- function(x) {
 data.frame(Pat=x$Pat[1],Phase=x$Phase[1],Q=x$Q[1],score = mean(x$Resp))
}

# about 20 sec
caScore <- by(df,list(Pat=df$Pat,Phase=df$Phase,Q=df$Q),FUN = score )

ca1 = do.call("rbind",caScore)
# Without lme4: 3 seconds
# With lme4: After 60 sec:
#Error: evaluation nested too deeply: infinite recursion /
#options(expressions=)?


-----------------------

platform i386-pc-mingw32
arch     i386
os       mingw32
system   i386, mingw32
status
major    2
minor    2.1
year     2005
month    12
day      20
svn rev  36812
language R

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: "infinite recursion" in do.call when lme4 loaded only

Peter Dalgaard
"Dieter Menne" <[hidden email]> writes:

> A larg program which worked with lme4/R about a year ago failed when I
> re-run it today. I reproduced the problem with the program below.
>
> -- When lme4 is not loaded, the program runs ok and fast enough
> -- When lme4 is loaded (but never used), the do.call fails
>    with infinite recursion after 60 seconds. Memory used increases
>    beyond bonds in task manager.
> -- I tested a few S3 based packages (MASS, nlme) and did not get
>    similar problems
>
> Current workaround: do lme4-processing in a separate program.

Looks like it conks out when the number of frames to rbind is bigger
than about 110.

Current releases have
> options("expressions")
$expressions
[1] 1000

It was 5000 for a while, but we found that it could overflow the C
stack on some systems. Since your example has 720 lines, I can't quite
rule out that that the problem was really there all the time.

However, it surely has to do with methods dispatch:

> system.time(do.call("rbind.data.frame",caScore))
[1] 0.99 0.00 0.99 0.00 0.00

which provides you with another workaround.

 

>
> ------
> #library(lme4) # uncomment this to see the problem
> np <- 12
> nq <- 20
> nph <- 3
> nrep <- 30
> grd <- expand.grid(Pat=as.factor(1:np),
>             Q=as.factor(1:nq),
>             Phase=as.factor(1:nph))
> df <- with (grd,
>   data.frame(Pat=Pat,Q=Q,Phase=Phase,Resp = rnorm(np*nq*nph*nrep)))
>
> score <- function(x) {
>  data.frame(Pat=x$Pat[1],Phase=x$Phase[1],Q=x$Q[1],score = mean(x$Resp))
> }
>
> # about 20 sec
> caScore <- by(df,list(Pat=df$Pat,Phase=df$Phase,Q=df$Q),FUN = score )
>
> ca1 = do.call("rbind",caScore)
> # Without lme4: 3 seconds
> # With lme4: After 60 sec:
> #Error: evaluation nested too deeply: infinite recursion /
> #options(expressions=)?
>
>
> -----------------------
>
> platform i386-pc-mingw32
> arch     i386
> os       mingw32
> system   i386, mingw32
> status
> major    2
> minor    2.1
> year     2005
> month    12
> day      20
> svn rev  36812
> language R
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>

--
   O__  ---- Peter Dalgaard             Ă˜ster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - ([hidden email])                  FAX: (+45) 35327907

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: "infinite recursion" in do.call when lme4 loaded only

Dieter Menne
Peter Dalgaard <p.dalgaard <at> biostat.ku.dk> writes:

> > A larg program which worked with lme4/R about a year ago failed when I
> > re-run it today. I reproduced the problem with the program below.

> > -- When lme4 is loaded (but never used), the do.call fails
> >    with infinite recursion after 60 seconds. Memory used increases
> >    beyond bonds in task manager.
>
> However, it surely has to do with methods dispatch:
>
> > system.time(do.call("rbind.data.frame",caScore))
> [1] 0.99 0.00 0.99 0.00 0.00
>
> which provides you with another workaround.

Peter, I had increased the optional value already, but I still don't understand
what this recursion overflow has to do with the lm4 loading.

Dieter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: "infinite recursion" in do.call when lme4 loaded only

Martin Maechler
>>>>> "Dieter" == Dieter Menne <[hidden email]>
>>>>>     on Thu, 12 Jan 2006 18:14:32 +0000 (UTC) writes:

    Dieter> Peter Dalgaard <p.dalgaard <at> biostat.ku.dk> writes:
    >> > A larg program which worked with lme4/R about a year ago failed when I
    >> > re-run it today. I reproduced the problem with the program below.

    >> > -- When lme4 is loaded (but never used), the do.call fails
    >> >    with infinite recursion after 60 seconds. Memory used increases
    >> >    beyond bonds in task manager.
    >>
    >> However, it surely has to do with methods dispatch:
    >>
    >> > system.time(do.call("rbind.data.frame",caScore))
    >> [1] 0.99 0.00 0.99 0.00 0.00
    >>
    >> which provides you with another workaround.

    Dieter> Peter, I had increased the optional value already, but I still don't understand
    Dieter> what this recursion overflow has to do with the lm4 loading.

Aahh, you've hit a secret ;-)  no, but a semi-hidden feature:
lme4 loads Matrix and Matrix  activates versions of rbind() and
cbind() which use rbind2/cbind2 which are S4 generics and
default methods that are slightly different than then the
original base rbind() and cbind().
This was a necessity since the original rbind(), cbind() have
first argument "...", i.e. an invalid signature for S4 method
dispatch.

This was in NEWS for R 2.2.0 :

    o Experimental versions of cbind() and rbind() in methods package,
        based on new generic function cbind2(x,y) and rbind2(). This will
        allow the equivalent of S4 methods for cbind() and rbind() ---
        currently only after an explicit activation call, see ?cbind2.

And 'Matrix' uses the activation call in its .OnLoad hook.
This is now getting much too technical to explain for R-help, so
if we want to go there, we should move this topic to R-devel,
and I'd like to do so, and will be glad if you can provide more
details on how exactly you're using rbind.

Martin Maechler,
ETH Zurich

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: [Rd] "infinite recursion" in do.call when lme4 loaded only

Martin Maechler
>>>>> "MM" == Martin Maechler <[hidden email]>
>>>>>     on Fri, 13 Jan 2006 12:24:51 +0100 writes:

>>>>> "Dieter" == Dieter Menne <[hidden email]>
>>>>>     on Thu, 12 Jan 2006 18:14:32 +0000 (UTC) writes:

    Dieter> Peter Dalgaard <p.dalgaard <at> biostat.ku.dk> writes:
    >>> > A larg program which worked with lme4/R about a year ago failed when I
    >>> > re-run it today. I reproduced the problem with the program below.

    >>> > -- When lme4 is loaded (but never used), the do.call fails
    >>> >    with infinite recursion after 60 seconds. Memory used increases
    >>> >    beyond bonds in task manager.
    >>>
    >>> However, it surely has to do with methods dispatch:
    >>>
    >>> > system.time(do.call("rbind.data.frame",caScore))
    >>> [1] 0.99 0.00 0.99 0.00 0.00
    >>>
    >>> which provides you with another workaround.

    Dieter> Peter, I had increased the optional value already, but I still don't understand
    Dieter> what this recursion overflow has to do with the lm4 loading.

    MM> Aahh, you've hit a secret ;-)  no, but a semi-hidden feature:
    MM> lme4 loads Matrix and Matrix  activates versions of rbind() and
    MM> cbind() which use rbind2/cbind2 which are S4 generics and
    MM> default methods that are slightly different than then the
    MM> original base rbind() and cbind().
    MM> This was a necessity since the original rbind(), cbind() have
    MM> first argument "...", i.e. an invalid signature for S4 method
    MM> dispatch.

    MM> This was in NEWS for R 2.2.0 :

    MM> o Experimental versions of cbind() and rbind() in methods package,
    MM> based on new generic function cbind2(x,y) and rbind2(). This will
    MM> allow the equivalent of S4 methods for cbind() and rbind() ---
    MM> currently only after an explicit activation call, see ?cbind2.

    MM> And 'Matrix' uses the activation call in its .OnLoad hook.
    MM> This is now getting much too technical to explain for R-help, so
    MM> if we want to go there, we should move this topic to R-devel,
    MM> and I'd like to do so, and will be glad if you can provide more
    MM> details on how exactly you're using rbind.

One thing -- very useful for you -- I forgot to add:

You can easily quickly revert the  "other cbind/rbind
activation" by using

    methods:::bind_activation(FALSE)

so you don't need to unload lme4 or Matrix,  and you can
reactivate them again after your special computation by

    methods:::bind_activation(on = TRUE)

Martin

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Rd] "infinite recursion" in do.call when lme4 loaded only

Dieter Menne
Martin Maechler <maechler <at> stat.math.ethz.ch> writes:

> One thing -- very useful for you -- I forgot to add:
>
> You can easily quickly revert the  "other cbind/rbind
> activation" by using
>
>     methods:::bind_activation(FALSE)
>
> so you don't need to unload lme4 or Matrix,  and you can
> reactivate them again after your special computation by
>
>     methods:::bind_activation(on = TRUE)

Do haut's di hi... Looks like we need a non-recursive replacement for
the elegant do.call("rbind"..) constructs.

Dieter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html