load huge image

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

load huge image

Gottfried Gruber
hello,

i have run around 65000 regressions and stored them in a list. then i stored
the session with save.image on my hard disk. the file is almost 1GB. when i
now want to load the image it took tons of time. even after 12h of loading it
was not done, although the saving was done fairly fast.

i fear i have to run the regressions again and store them in a database ...

can i load this file? any suggestions?

thanks & bets regards,
gg
--
---------------------------------------------------
Gottfried Gruber
mailto:[hidden email]
www: http://gogo.sehrsupa.net

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: load huge image

Martin Maechler
>>>>> "Gottfried" == Gottfried Gruber <[hidden email]>
>>>>>     on Sun, 26 Mar 2006 10:27:35 +0200 writes:

    Gottfried> hello, i have run around 65000 regressions and
    Gottfried> stored them in a list. then i stored the session
    Gottfried> with save.image on my hard disk. the file is
    Gottfried> almost 1GB. when i now want to load the image it
    Gottfried> took tons of time. even after 12h of loading it
    Gottfried> was not done, although the saving was done fairly
    Gottfried> fast.

I'm sure it takes so lang because you (i.e. R) run out of RAM
and the machine starts to swap.

Try to get access to a Linux (or other Unix-alike) machine with
a 64-bit version of R and about 8 GB of RAM (maybe 4 GB is
already sufficient). I guess then you should be able to read it
much more quickly.

For 65000 regressions, do you need more than the estimated
coefficients or -- a bit more informatively -- the

  coef(summary( <lm-fit> ))  result ?

If you had only saved these coefficient matrices, I'm sure you'd
have need **much** less memory.



    Gottfried> i fear i have to run the regressions again and
    Gottfried> store them in a database ...

or really store what you need instead of everything ...

    Gottfried> can i load this file? any suggestions?

    Gottfried> thanks & bets regards, gg --
    Gottfried> ---------------------------------------------------
    Gottfried> Gottfried Gruber
    Gottfried> mailto:[hidden email] www:
    Gottfried> http://gogo.sehrsupa.net

    Gottfried> ______________________________________________
    Gottfried> [hidden email] mailing list
    Gottfried> https://stat.ethz.ch/mailman/listinfo/r-help
    Gottfried> PLEASE do read the posting guide!
    Gottfried> http://www.R-project.org/posting-guide.html

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: load huge image

Henrik Bengtsson-2
On 3/27/06, Martin Maechler <[hidden email]> wrote:

> >>>>> "Gottfried" == Gottfried Gruber <[hidden email]>
> >>>>>     on Sun, 26 Mar 2006 10:27:35 +0200 writes:
>
>     Gottfried> hello, i have run around 65000 regressions and
>     Gottfried> stored them in a list. then i stored the session
>     Gottfried> with save.image on my hard disk. the file is
>     Gottfried> almost 1GB. when i now want to load the image it
>     Gottfried> took tons of time. even after 12h of loading it
>     Gottfried> was not done, although the saving was done fairly
>     Gottfried> fast.
>
> I'm sure it takes so lang because you (i.e. R) run out of RAM
> and the machine starts to swap.
>
> Try to get access to a Linux (or other Unix-alike) machine with
> a 64-bit version of R and about 8 GB of RAM (maybe 4 GB is
> already sufficient). I guess then you should be able to read it
> much more quickly.
>
> For 65000 regressions, do you need more than the estimated
> coefficients or -- a bit more informatively -- the
>
>   coef(summary( <lm-fit> ))  result ?
>
> If you had only saved these coefficient matrices, I'm sure you'd
> have need **much** less memory.
>
>
>     Gottfried> i fear i have to run the regressions again and
>     Gottfried> store them in a database ...
>
> or really store what you need instead of everything ...
>
>     Gottfried> can i load this file? any suggestions?

Do you need to have all of them in memory at once?  Instead of using
save.image() can't you use save()/load() on each of the regression
fits?  You can name the files using sprintf("regression%05d.Rdata",
idx) or similar.

Also, as Martin says, "fit" objects contains a lot of information that
you might not need; remove these before saving by setting the elements
you don't want to NULL.

/Henrik

>     Gottfried> thanks & bets regards, gg --
>     Gottfried> ---------------------------------------------------
>     Gottfried> Gottfried Gruber
>     Gottfried> mailto:[hidden email] www:
>     Gottfried> http://gogo.sehrsupa.net
>
>     Gottfried> ______________________________________________
>     Gottfried> [hidden email] mailing list
>     Gottfried> https://stat.ethz.ch/mailman/listinfo/r-help
>     Gottfried> PLEASE do read the posting guide!
>     Gottfried> http://www.R-project.org/posting-guide.html
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>
>


--
Henrik Bengtsson
Mobile: +46 708 909208 (+2h UTC)

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html