Error: cannot allocate vector of size 3.0 Gb

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Error: cannot allocate vector of size 3.0 Gb

John-88
Hello,

I have read recent posts on this topic (Dr. Ronnen Levinson's Monday 02:39:55
pm), but before I install a 64 bit system, and purchase more RAM, I want to
make sure I understand this interesting issue.

I was attempting to plot a data frame containing Dow Jones stock information:

> length(Date)
[1] 19957
> length(Close)
[1] 19957
> head(DowJones)
      Date     Open     High      Low    Close     Volume Adj.Close
1 03/26/08 12531.79 12531.79 12309.62 12422.86 4055670000  12422.86
2 03/25/08 12547.34 12639.82 12397.62 12532.60 4145120000  12532.60
3 03/24/08 12361.97 12687.61 12346.17 12548.64 4499000000  12548.64
4 03/20/08 12102.43 12434.34 12024.68 12361.32 2078450000  12361.32
5 03/19/08 12391.52 12525.19 12077.27 12099.66 1203830000  12099.66
6 03/18/08 11975.92 12411.63 11975.92 12392.66 1263390000  12392.66
> tail(DowJones)
          Date   Open   High    Low  Close  Volume Adj.Close
19952 10/08/28 240.17 243.33 237.72 239.55 3940000    239.55
19953 10/05/28 240.00 243.08 238.22 240.44 4360000    240.44
19954 10/04/28 237.75 242.53 237.72 240.00 4330000    240.00
19955 10/03/28 238.14 239.14 233.60 237.75 4060000    237.75
19956 10/02/28 240.01 241.54 235.42 238.14 3850000    238.14
19957 10/01/28 239.43 242.46 238.24 240.01 3500000    240.01
> plot(Date, Close)
Error: cannot allocate vector of size 3.0 Gb
>                                              


I currently have installed R 2.60, from source, onto Linux SuSE 10.2. I
believe it is a 32 bit install:

> dmesg | grep bit
> ...
> hpet0: 3 32-bit timers, 25000000 Hz

I have 1GB of RAM, with circa 883.61 not allocated to hardware after boot.
I naively thought 19957 rows of data would be easily handled by a fairly new
computer. R was able to cease execution, and place control back onto its
interpreter, while a Chart attempt in OpenOffice locks up OpenOffice.

So,  will installation of a 64 bit Operating System, and at least an
additional 1GB of RAM  suffice to generate memory space for a 3.0 Gb vector,
in R?

Much appreciated, time to buy a new computer?

John

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error: cannot allocate vector of size 3.0 Gb

Liaw, Andy
Looks like you attach() the data frame before you try t/o plot.  Note
that in the "Details" section of ?attach, it says:

"... The database is not actually attached. Rather, a new environment is
created on the search path and the elements of a list (including columns
of a data frame) or objects in a save file or an environment are copied
into the new environment. ..."

so it's probably a bad idea to do the attach().  Have you tried:

with(DowJones, plot(Date, Close))

?

About 4 years ago I've posted an example of creating a matrix that took
up >6GB, on a dual Opteron w/ 16GB ram running 64-bit SuSE and 64-bit R.
So yes, it's possible.  Nowadays we do it nearly on a daily basis.

Andy

From: John

>
> Hello,
>
> I have read recent posts on this topic (Dr. Ronnen Levinson's
> Monday 02:39:55
> pm), but before I install a 64 bit system, and purchase more
> RAM, I want to
> make sure I understand this interesting issue.
>
> I was attempting to plot a data frame containing Dow Jones
> stock information:
>
> > length(Date)
> [1] 19957
> > length(Close)
> [1] 19957
> > head(DowJones)
>       Date     Open     High      Low    Close     Volume Adj.Close
> 1 03/26/08 12531.79 12531.79 12309.62 12422.86 4055670000  12422.86
> 2 03/25/08 12547.34 12639.82 12397.62 12532.60 4145120000  12532.60
> 3 03/24/08 12361.97 12687.61 12346.17 12548.64 4499000000  12548.64
> 4 03/20/08 12102.43 12434.34 12024.68 12361.32 2078450000  12361.32
> 5 03/19/08 12391.52 12525.19 12077.27 12099.66 1203830000  12099.66
> 6 03/18/08 11975.92 12411.63 11975.92 12392.66 1263390000  12392.66
> > tail(DowJones)
>           Date   Open   High    Low  Close  Volume Adj.Close
> 19952 10/08/28 240.17 243.33 237.72 239.55 3940000    239.55
> 19953 10/05/28 240.00 243.08 238.22 240.44 4360000    240.44
> 19954 10/04/28 237.75 242.53 237.72 240.00 4330000    240.00
> 19955 10/03/28 238.14 239.14 233.60 237.75 4060000    237.75
> 19956 10/02/28 240.01 241.54 235.42 238.14 3850000    238.14
> 19957 10/01/28 239.43 242.46 238.24 240.01 3500000    240.01
> > plot(Date, Close)
> Error: cannot allocate vector of size 3.0 Gb
> >                                              
>
>
> I currently have installed R 2.60, from source, onto Linux
> SuSE 10.2. I
> believe it is a 32 bit install:
>
> > dmesg | grep bit
> > ...
> > hpet0: 3 32-bit timers, 25000000 Hz
>
> I have 1GB of RAM, with circa 883.61 not allocated to
> hardware after boot.
> I naively thought 19957 rows of data would be easily handled
> by a fairly new
> computer. R was able to cease execution, and place control
> back onto its
> interpreter, while a Chart attempt in OpenOffice locks up OpenOffice.
>
> So,  will installation of a 64 bit Operating System, and at least an
> additional 1GB of RAM  suffice to generate memory space for a
> 3.0 Gb vector,
> in R?
>
> Much appreciated, time to buy a new computer?
>
> John
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>


------------------------------------------------------------------------------
Notice:  This e-mail message, together with any attachme...{{dropped:15}}

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error: cannot allocate vector of size 3.0 Gb

Daniel Nordlund
In reply to this post by John-88
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf
> Of John
> Sent: Friday, March 28, 2008 12:04 PM
> To: [hidden email]
> Subject: [R] Error: cannot allocate vector of size 3.0 Gb
>
> Hello,
>
> I have read recent posts on this topic (Dr. Ronnen Levinson's Monday 02:39:55
> pm), but before I install a 64 bit system, and purchase more RAM, I want to
> make sure I understand this interesting issue.
>
> I was attempting to plot a data frame containing Dow Jones stock information:
>
> > length(Date)
> [1] 19957
> > length(Close)
> [1] 19957
> > head(DowJones)
>       Date     Open     High      Low    Close     Volume Adj.Close
> 1 03/26/08 12531.79 12531.79 12309.62 12422.86 4055670000  12422.86
> 2 03/25/08 12547.34 12639.82 12397.62 12532.60 4145120000  12532.60
> 3 03/24/08 12361.97 12687.61 12346.17 12548.64 4499000000  12548.64
> 4 03/20/08 12102.43 12434.34 12024.68 12361.32 2078450000  12361.32
> 5 03/19/08 12391.52 12525.19 12077.27 12099.66 1203830000  12099.66
> 6 03/18/08 11975.92 12411.63 11975.92 12392.66 1263390000  12392.66
> > tail(DowJones)
>           Date   Open   High    Low  Close  Volume Adj.Close
> 19952 10/08/28 240.17 243.33 237.72 239.55 3940000    239.55
> 19953 10/05/28 240.00 243.08 238.22 240.44 4360000    240.44
> 19954 10/04/28 237.75 242.53 237.72 240.00 4330000    240.00
> 19955 10/03/28 238.14 239.14 233.60 237.75 4060000    237.75
> 19956 10/02/28 240.01 241.54 235.42 238.14 3850000    238.14
> 19957 10/01/28 239.43 242.46 238.24 240.01 3500000    240.01
> > plot(Date, Close)
> Error: cannot allocate vector of size 3.0 Gb
> >
>
>
> I currently have installed R 2.60, from source, onto Linux SuSE 10.2. I
> believe it is a 32 bit install:
>
> > dmesg | grep bit
> > ...
> > hpet0: 3 32-bit timers, 25000000 Hz
>
> I have 1GB of RAM, with circa 883.61 not allocated to hardware after boot.
> I naively thought 19957 rows of data would be easily handled by a fairly new
> computer. R was able to cease execution, and place control back onto its
> interpreter, while a Chart attempt in OpenOffice locks up OpenOffice.
>
> So,  will installation of a 64 bit Operating System, and at least an
> additional 1GB of RAM  suffice to generate memory space for a 3.0 Gb vector,
> in R?
>

John,

Am I missing something here?  You have one 1 GB of RAM, you want to get a second 1 GB of RAM, and you want to know if you will be able to create a 3 GB vector?  With or without a 64bit OS, I don't think you will be successful.  In addition to total memory available, I believe you need to have sufficient contiguous memory to allocate an object.  If memory is fragmented, you won't be able to create the vector, even if the total free space is sufficient.  If I am incorrect, I'm sure someone will be along to correct me shortly.

Hope this is helpful,

Dan

Daniel Nordlund
Bothell, WA

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error: cannot allocate vector of size 3.0 Gb

Uwe Ligges-3
In reply to this post by John-88
Can you make the data.frame available somewhere. Actually, I am
surprised it need that huge amount of memory to do the plot.

Best,
Uwe Ligges

John wrote:

> Hello,
>
> I have read recent posts on this topic (Dr. Ronnen Levinson's Monday 02:39:55
> pm), but before I install a 64 bit system, and purchase more RAM, I want to
> make sure I understand this interesting issue.
>
> I was attempting to plot a data frame containing Dow Jones stock information:
>
>> length(Date)
> [1] 19957
>> length(Close)
> [1] 19957
>> head(DowJones)
>       Date     Open     High      Low    Close     Volume Adj.Close
> 1 03/26/08 12531.79 12531.79 12309.62 12422.86 4055670000  12422.86
> 2 03/25/08 12547.34 12639.82 12397.62 12532.60 4145120000  12532.60
> 3 03/24/08 12361.97 12687.61 12346.17 12548.64 4499000000  12548.64
> 4 03/20/08 12102.43 12434.34 12024.68 12361.32 2078450000  12361.32
> 5 03/19/08 12391.52 12525.19 12077.27 12099.66 1203830000  12099.66
> 6 03/18/08 11975.92 12411.63 11975.92 12392.66 1263390000  12392.66
>> tail(DowJones)
>           Date   Open   High    Low  Close  Volume Adj.Close
> 19952 10/08/28 240.17 243.33 237.72 239.55 3940000    239.55
> 19953 10/05/28 240.00 243.08 238.22 240.44 4360000    240.44
> 19954 10/04/28 237.75 242.53 237.72 240.00 4330000    240.00
> 19955 10/03/28 238.14 239.14 233.60 237.75 4060000    237.75
> 19956 10/02/28 240.01 241.54 235.42 238.14 3850000    238.14
> 19957 10/01/28 239.43 242.46 238.24 240.01 3500000    240.01
>> plot(Date, Close)
> Error: cannot allocate vector of size 3.0 Gb
>>                                              
>
>
> I currently have installed R 2.60, from source, onto Linux SuSE 10.2. I
> believe it is a 32 bit install:
>
>> dmesg | grep bit
>> ...
>> hpet0: 3 32-bit timers, 25000000 Hz
>
> I have 1GB of RAM, with circa 883.61 not allocated to hardware after boot.
> I naively thought 19957 rows of data would be easily handled by a fairly new
> computer. R was able to cease execution, and place control back onto its
> interpreter, while a Chart attempt in OpenOffice locks up OpenOffice.
>
> So,  will installation of a 64 bit Operating System, and at least an
> additional 1GB of RAM  suffice to generate memory space for a 3.0 Gb vector,
> in R?
>
> Much appreciated, time to buy a new computer?
>
> John
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error: cannot allocate vector of size 3.0 Gb

John-88
In reply to this post by Daniel Nordlund
On Friday 28 March 2008 14:28, Daniel Nordlund wrote:

> > -----Original Message-----
> > From: [hidden email] [mailto:[hidden email]]
> > On Behalf Of John
> > Sent: Friday, March 28, 2008 12:04 PM
> > To: [hidden email]
> > Subject: [R] Error: cannot allocate vector of size 3.0 Gb
> >
> > Hello,
> >
> > I have read recent posts on this topic (Dr. Ronnen Levinson's Monday
> > 02:39:55 pm), but before I install a 64 bit system, and purchase more
> > RAM, I want to make sure I understand this interesting issue.
> >
> > I was attempting to plot a data frame containing Dow Jones stock
information:

> > > length(Date)
> >
> > [1] 19957
> >
> > > length(Close)
> >
> > [1] 19957
> >
> > > head(DowJones)
> >
> >       Date     Open     High      Low    Close     Volume Adj.Close
> > 1 03/26/08 12531.79 12531.79 12309.62 12422.86 4055670000  12422.86
> > 2 03/25/08 12547.34 12639.82 12397.62 12532.60 4145120000  12532.60
> > 3 03/24/08 12361.97 12687.61 12346.17 12548.64 4499000000  12548.64
> > 4 03/20/08 12102.43 12434.34 12024.68 12361.32 2078450000  12361.32
> > 5 03/19/08 12391.52 12525.19 12077.27 12099.66 1203830000  12099.66
> > 6 03/18/08 11975.92 12411.63 11975.92 12392.66 1263390000  12392.66
> >
> > > tail(DowJones)
> >
> >           Date   Open   High    Low  Close  Volume Adj.Close
> > 19952 10/08/28 240.17 243.33 237.72 239.55 3940000    239.55
> > 19953 10/05/28 240.00 243.08 238.22 240.44 4360000    240.44
> > 19954 10/04/28 237.75 242.53 237.72 240.00 4330000    240.00
> > 19955 10/03/28 238.14 239.14 233.60 237.75 4060000    237.75
> > 19956 10/02/28 240.01 241.54 235.42 238.14 3850000    238.14
> > 19957 10/01/28 239.43 242.46 238.24 240.01 3500000    240.01
> >
> > > plot(Date, Close)
> >
> > Error: cannot allocate vector of size 3.0 Gb
> >
> >
> >
> > I currently have installed R 2.60, from source, onto Linux SuSE 10.2. I
> >
> > believe it is a 32 bit install:
> > > dmesg | grep bit
> > > ...
> > > hpet0: 3 32-bit timers, 25000000 Hz
> >
> > I have 1GB of RAM, with circa 883.61 not allocated to hardware after
> > boot. I naively thought 19957 rows of data would be easily handled by a
> > fairly new computer. R was able to cease execution, and place control
> > back onto its interpreter, while a Chart attempt in OpenOffice locks up
> > OpenOffice.
> >
> > So,  will installation of a 64 bit Operating System, and at least an
> > additional 1GB of RAM  suffice to generate memory space for a 3.0 Gb
> > vector, in R?
>
> John,
>
> Am I missing something here?  You have one 1 GB of RAM, you want to get a
> second 1 GB of RAM, and you want to know if you will be able to create a 3
> GB vector?  With or without a 64bit OS, I don't think you will be
> successful.  In addition to total memory available, I believe you need to
> have sufficient contiguous memory to allocate an object.  If memory is
> fragmented, you won't be able to create the vector, even if the total free
> space is sufficient.  If I am incorrect, I'm sure someone will be along to
> correct me shortly.
>
> Hope this is helpful,
>
> Dan
>
> Daniel Nordlund
> Bothell, WA
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html and provide commented, minimal,
> self-contained, reproducible code.


Hi Daniel,

Thank you for the information. It has saved me from needlessly installing a
64-bit linux instance. In the short term, I'll try and generate multiple
graphs, side by side. And purchase more RAM. A lot more.

Enjoying the notion of multiple plots in R,

John

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Error: cannot allocate vector of size 3.0 Gb

Uwe Ligges-3
Dear John,

note that you "Date" is a factor rather than some date object.
If you convert it to some date object, just a few megabytes will suffice!

Best wishes,
Uwe




John wrote:

> On Friday 28 March 2008 14:28, Daniel Nordlund wrote:
>>> -----Original Message-----
>>> From: [hidden email] [mailto:[hidden email]]
>>> On Behalf Of John
>>> Sent: Friday, March 28, 2008 12:04 PM
>>> To: [hidden email]
>>> Subject: [R] Error: cannot allocate vector of size 3.0 Gb
>>>
>>> Hello,
>>>
>>> I have read recent posts on this topic (Dr. Ronnen Levinson's Monday
>>> 02:39:55 pm), but before I install a 64 bit system, and purchase more
>>> RAM, I want to make sure I understand this interesting issue.
>>>
>>> I was attempting to plot a data frame containing Dow Jones stock
> information:
>>>> length(Date)
>>> [1] 19957
>>>
>>>> length(Close)
>>> [1] 19957
>>>
>>>> head(DowJones)
>>>       Date     Open     High      Low    Close     Volume Adj.Close
>>> 1 03/26/08 12531.79 12531.79 12309.62 12422.86 4055670000  12422.86
>>> 2 03/25/08 12547.34 12639.82 12397.62 12532.60 4145120000  12532.60
>>> 3 03/24/08 12361.97 12687.61 12346.17 12548.64 4499000000  12548.64
>>> 4 03/20/08 12102.43 12434.34 12024.68 12361.32 2078450000  12361.32
>>> 5 03/19/08 12391.52 12525.19 12077.27 12099.66 1203830000  12099.66
>>> 6 03/18/08 11975.92 12411.63 11975.92 12392.66 1263390000  12392.66
>>>
>>>> tail(DowJones)
>>>           Date   Open   High    Low  Close  Volume Adj.Close
>>> 19952 10/08/28 240.17 243.33 237.72 239.55 3940000    239.55
>>> 19953 10/05/28 240.00 243.08 238.22 240.44 4360000    240.44
>>> 19954 10/04/28 237.75 242.53 237.72 240.00 4330000    240.00
>>> 19955 10/03/28 238.14 239.14 233.60 237.75 4060000    237.75
>>> 19956 10/02/28 240.01 241.54 235.42 238.14 3850000    238.14
>>> 19957 10/01/28 239.43 242.46 238.24 240.01 3500000    240.01
>>>
>>>> plot(Date, Close)
>>> Error: cannot allocate vector of size 3.0 Gb
>>>
>>>
>>>
>>> I currently have installed R 2.60, from source, onto Linux SuSE 10.2. I
>>>
>>> believe it is a 32 bit install:
>>>> dmesg | grep bit
>>>> ...
>>>> hpet0: 3 32-bit timers, 25000000 Hz
>>> I have 1GB of RAM, with circa 883.61 not allocated to hardware after
>>> boot. I naively thought 19957 rows of data would be easily handled by a
>>> fairly new computer. R was able to cease execution, and place control
>>> back onto its interpreter, while a Chart attempt in OpenOffice locks up
>>> OpenOffice.
>>>
>>> So,  will installation of a 64 bit Operating System, and at least an
>>> additional 1GB of RAM  suffice to generate memory space for a 3.0 Gb
>>> vector, in R?
>> John,
>>
>> Am I missing something here?  You have one 1 GB of RAM, you want to get a
>> second 1 GB of RAM, and you want to know if you will be able to create a 3
>> GB vector?  With or without a 64bit OS, I don't think you will be
>> successful.  In addition to total memory available, I believe you need to
>> have sufficient contiguous memory to allocate an object.  If memory is
>> fragmented, you won't be able to create the vector, even if the total free
>> space is sufficient.  If I am incorrect, I'm sure someone will be along to
>> correct me shortly.
>>
>> Hope this is helpful,
>>
>> Dan
>>
>> Daniel Nordlund
>> Bothell, WA
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html and provide commented, minimal,
>> self-contained, reproducible code.
>
>
> Hi Daniel,
>
> Thank you for the information. It has saved me from needlessly installing a
> 64-bit linux instance. In the short term, I'll try and generate multiple
> graphs, side by side. And purchase more RAM. A lot more.
>
> Enjoying the notion of multiple plots in R,
>
> John
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.