Quantcast

Error: cannot allocate vector of size

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Error: cannot allocate vector of size

xin123620
This post was updated on .
Dear R Users,

I was using R to import five years traffic data, but the error always come up as shown below. The data frame contains 12 columns and unknown number of records. I changed the R memory sets to 4000, but still doesn't work. This R is 32bit. This computer is Windows XP, 32bit Intel(R) Core(TM) E8400@3.GHz, 2.99GHz, 2.95GB of RAM.  Would you have any ideas that how I should deal with this situation?  Many thanks for any hints.

>wim<-sqlQuery(channel,qry)
>wim
Error: cannot allocate vector of size 24.0 Mb
In addition: There were 16 warnings (use warnings() to see them)

>memory.size(max=T)
[1] 1535.62
>memory.size(max=F)
[1]16.24
>gc()
used (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells  141802  3.8    6160587  164.6   5130092  137.0
Vcells 1100479  8.4  200894152 1532.8 184314496 1406.3
I changed the r memory limit

>memory.limit(4000)
[1] 4000
>wim<-sqlQuery(channel,qry)
Garbage collection 98 = 3+1+94 (level 2) ...
47.8 Mbytes of cons cells used (29%)
167.8 Mbytes of vectors used (11%)
Garbage collection 99 = 4+1+94 (level 0) ...
136.9 Mbytes of cons cells used (83%)
798.8 Mbytes of vectors used (55%)
Garbage collection 100 = 4+1+95 (level 2) ...
136.9 Mbytes of cons cells used (79%)
1002.0 Mbytes of vectors used (65%)
Garbage collection 101 = 4+1+96 (level 2) ...
136.9 Mbytes of cons cells used (75%)
1170.3 Mbytes of vectors used (73%)
Garbage collection 102 = 4+1+97 (level 2) ...
136.9 Mbytes of cons cells used (71%)
1074.1 Mbytes of vectors used (67%)
Garbage collection 103 = 4+1+98 (level 2) ...
136.9 Mbytes of cons cells used (67%)
1122.2 Mbytes of vectors used (66%)
Garbage collection 104 = 4+1+99 (level 2) ...
136.9 Mbytes of cons cells used (67%)
1170.3 Mbytes of vectors used (66%)
Garbage collection 105 = 4+1+100 (level 2) ...
136.9 Mbytes of cons cells used (67%)
1266.5 Mbytes of vectors used (68%)
Garbage collection 106 = 4+1+101 (level 2) ...
136.9 Mbytes of cons cells used (67%)
1507.0 Mbytes of vectors used (77%)
Error: cannot allocate vector of size 24.0 Mb
In addition: Warning message:
closing unused RODBC handle 1

>sessionInfo()
R version 2.12.2 (2011-02-25)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base    

other attached packages:
[1] RODBC_1.3-2

Best,
Chengxin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error: cannot allocate vector of size

Jeff Newmiller
A) You haven't mentioned your OS which indicates you haven't followed the posting guide noted at the bottom of each email.

B) You cannot load an "unknown" number of rows... although you may not specify the number, it is finite and its value can be determined for the purposes of debugging your immediate problem by examining your data.in its external storage. Please investigate and inform us.
---------------------------------------------------------------------------
Jeff Newmiller The ..... ..... Go Live...
DCN:<[hidden email]> Basics: ##.#. ##.#. Live Go...
Live: OO#.. Dead: OO#.. Playing
Research Engineer (Solar/Batteries O.O#. #.O#. with
/Software/Embedded Controllers) .OO#. .OO#. rocks...1k
---------------------------------------------------------------------------
Sent from my phone. Please excuse my brevity.

xin123620 <[hidden email]> wrote:

Dear R Users,

I was using R to import five years traffic data, but the error always come
up as shown below. The data frame contains 12 columns and unknown number of
records. Would you have any ideas that how I should deal with this
situation?  Many thanks for any hints.

>wim<-sqlQuery(channel,qry)
Error: cannot allocate vector of size 24.0 Mb
In addition: There were 16 warnings (use warnings() to see them)

>memory.size(max=T)
[1] 1535.62
>memory.size(max=F)
[1]16.24
>gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 141802 3.8 6160587 164.6 5130092 137.0
Vcells 1100479 8.4 200894152 1532.8 184314496 1406.3


Best,
Chengxin


--
View this message in context: http://r.789695.n4.nabble.com/Error-cannot-allocate-vector-of-size-tp3629384p3629384.html
Sent from the R help mailing list archive at Nabble.com.

_____________________________________________

[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


        [[alternative HTML version deleted]]


______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error: cannot allocate vector of size

xin123620
This post was updated on .
Thank you Jeff. You are absolutely right. I just edited posting and added the R and computer info in: R is 32 bit; computer is his computer is Windows XP, 32bit Intel(R) Core(TM) E8400@3.GHz, 2.99GHz, 2.95GB of RAM.

The data I am trying to retrieve is through postgre from a university server. The total records are roughly 31,518,000.

Would you give some hints please? Thank you very much.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error: cannot allocate vector of size

jholtman
select count(*) from yourData

On Tue, Jun 28, 2011 at 3:07 PM, xin123620 <[hidden email]> wrote:

> Thank you Jeff. You are absolutely right. I just edited the R and computer
> info in: R is 32 bit; computer is his computer is Windows XP, 32bit Intel(R)
> Core(TM) [hidden email], 2.99GHz, 2.95GB of RAM.
>
> The data I am trying to retrieve is through postgre from a university
> server. I checked the postgre for a while but still wouldn't find the number
> of total records. Would you give some hints please? Thank you very much.
>
> --
> View this message in context: http://r.789695.n4.nabble.com/Error-cannot-allocate-vector-of-size-tp3629384p3631168.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



--
Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error: cannot allocate vector of size

xin123620
This post was updated on .
Thank you Jholtman.
Now count is 46,001,902. I was trying to retrieve one-year data, but I still receive the following message:
"Error: cannot allocate vector of size 64.0 Mb"

So far I've tried: change the visual memory in Windown XP; change the R memory limit: review the R-forum postings. But I still cann't solve this problem. Would you help me figure it out ? Thanks in advance.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error: cannot allocate vector of size

jholtman
Assuming that your column are numeric, you would need 4GB of memory
just to store one copy of the object.  If this is 5 years, then you
would need almost 1GB for a copy, but the processing probably will use
up twice as much as it is processing.  Try reading a month's worth and
see how much you use.  In problems like this, you creep up on it; you
just try to jump in all at once.  Do it in increments and watch the
usage.

On Tue, Jun 28, 2011 at 4:18 PM, xin123620 <[hidden email]> wrote:

> Thank you Jholtman.
> Now count is 46001902. I was trying to retrieve one-year data, but I still
> receive the following message:
> "Error: cannot allocate vector of size 64.0 Mb"
>
> --
> View this message in context: http://r.789695.n4.nabble.com/Error-cannot-allocate-vector-of-size-tp3629384p3631354.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



--
Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error: cannot allocate vector of size

xin123620
Thank you for replying. when I've tried to run the R syntax in a 64 bit computer,the problem is solved.  Thank you for helping out. I totally agree your advice.
I would like to answer all your questions in case other people meet the same problem. The data contains one timestamp column with time zone, one integer column, and 12 Boolean columns. I tried to run one month sample,
before
>memory.limit()
[1] 3583
> memory.size(max=F)
[1] 156.82
> memory.size(max=T)
[1] 241.06
After
> memory.limit()
[1] 3583
> memory.size(max=F)
[1] 199.97
> memory.size(max=T)
[1] 241.56



Loading...