This post was updated on .
Dear R Users,
I was using R to import five years traffic data, but the error always come up as shown below. The data frame contains 12 columns and unknown number of records. I changed the R memory sets to 4000, but still doesn't work. This R is 32bit. This computer is Windows XP, 32bit Intel(R) Core(TM) E8400@3.GHz, 2.99GHz, 2.95GB of RAM. Would you have any ideas that how I should deal with this situation? Many thanks for any hints. >wim<-sqlQuery(channel,qry) >wim Error: cannot allocate vector of size 24.0 Mb In addition: There were 16 warnings (use warnings() to see them) >memory.size(max=T) [1] 1535.62 >memory.size(max=F) [1]16.24 >gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells 141802 3.8 6160587 164.6 5130092 137.0 Vcells 1100479 8.4 200894152 1532.8 184314496 1406.3 I changed the r memory limit >memory.limit(4000) [1] 4000 >wim<-sqlQuery(channel,qry) Garbage collection 98 = 3+1+94 (level 2) ... 47.8 Mbytes of cons cells used (29%) 167.8 Mbytes of vectors used (11%) Garbage collection 99 = 4+1+94 (level 0) ... 136.9 Mbytes of cons cells used (83%) 798.8 Mbytes of vectors used (55%) Garbage collection 100 = 4+1+95 (level 2) ... 136.9 Mbytes of cons cells used (79%) 1002.0 Mbytes of vectors used (65%) Garbage collection 101 = 4+1+96 (level 2) ... 136.9 Mbytes of cons cells used (75%) 1170.3 Mbytes of vectors used (73%) Garbage collection 102 = 4+1+97 (level 2) ... 136.9 Mbytes of cons cells used (71%) 1074.1 Mbytes of vectors used (67%) Garbage collection 103 = 4+1+98 (level 2) ... 136.9 Mbytes of cons cells used (67%) 1122.2 Mbytes of vectors used (66%) Garbage collection 104 = 4+1+99 (level 2) ... 136.9 Mbytes of cons cells used (67%) 1170.3 Mbytes of vectors used (66%) Garbage collection 105 = 4+1+100 (level 2) ... 136.9 Mbytes of cons cells used (67%) 1266.5 Mbytes of vectors used (68%) Garbage collection 106 = 4+1+101 (level 2) ... 136.9 Mbytes of cons cells used (67%) 1507.0 Mbytes of vectors used (77%) Error: cannot allocate vector of size 24.0 Mb In addition: Warning message: closing unused RODBC handle 1 >sessionInfo() R version 2.12.2 (2011-02-25) Platform: i386-pc-mingw32/i386 (32-bit) locale: [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 LC_NUMERIC=C LC_TIME=English_United States.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] RODBC_1.3-2 Best, Chengxin |
A) You haven't mentioned your OS which indicates you haven't followed the posting guide noted at the bottom of each email.
B) You cannot load an "unknown" number of rows... although you may not specify the number, it is finite and its value can be determined for the purposes of debugging your immediate problem by examining your data.in its external storage. Please investigate and inform us. --------------------------------------------------------------------------- Jeff Newmiller The ..... ..... Go Live... DCN:<[hidden email]> Basics: ##.#. ##.#. Live Go... Live: OO#.. Dead: OO#.. Playing Research Engineer (Solar/Batteries O.O#. #.O#. with /Software/Embedded Controllers) .OO#. .OO#. rocks...1k --------------------------------------------------------------------------- Sent from my phone. Please excuse my brevity. xin123620 <[hidden email]> wrote: Dear R Users, I was using R to import five years traffic data, but the error always come up as shown below. The data frame contains 12 columns and unknown number of records. Would you have any ideas that how I should deal with this situation?ã Many thanks for any hints. >wim<-sqlQuery(channel,qry) Error: cannot allocate vector of size 24.0 Mb In addition: There were 16 warnings (use warnings() to see them) >memory.size(max=T) [1] 1535.62 >memory.size(max=F) [1]16.24 >gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells 141802 3.8 6160587 164.6 5130092 137.0 Vcells 1100479 8.4 200894152 1532.8 184314496 1406.3 Best, Chengxin -- View this message in context: http://r.789695.n4.nabble.com/Error-cannot-allocate-vector-of-size-tp3629384p3629384.html Sent from the R help mailing list archive at Nabble.com. _____________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] ______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. |
This post was updated on .
Thank you Jeff. You are absolutely right. I just edited posting and added the R and computer info in: R is 32 bit; computer is his computer is Windows XP, 32bit Intel(R) Core(TM) E8400@3.GHz, 2.99GHz, 2.95GB of RAM.
The data I am trying to retrieve is through postgre from a university server. The total records are roughly 31,518,000. Would you give some hints please? Thank you very much. |
select count(*) from yourData
On Tue, Jun 28, 2011 at 3:07 PM, xin123620 <[hidden email]> wrote: > Thank you Jeff. You are absolutely right. I just edited the R and computer > info in: R is 32 bit; computer is his computer is Windows XP, 32bit Intel(R) > Core(TM) [hidden email], 2.99GHz, 2.95GB of RAM. > > The data I am trying to retrieve is through postgre from a university > server. I checked the postgre for a while but still wouldn't find the number > of total records. Would you give some hints please? Thank you very much. > > -- > View this message in context: http://r.789695.n4.nabble.com/Error-cannot-allocate-vector-of-size-tp3629384p3631168.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > [hidden email] mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? ______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. |
This post was updated on .
Thank you Jholtman.
Now count is 46,001,902. I was trying to retrieve one-year data, but I still receive the following message: "Error: cannot allocate vector of size 64.0 Mb" So far I've tried: change the visual memory in Windown XP; change the R memory limit: review the R-forum postings. But I still cann't solve this problem. Would you help me figure it out ? Thanks in advance. |
Assuming that your column are numeric, you would need 4GB of memory
just to store one copy of the object. If this is 5 years, then you would need almost 1GB for a copy, but the processing probably will use up twice as much as it is processing. Try reading a month's worth and see how much you use. In problems like this, you creep up on it; you just try to jump in all at once. Do it in increments and watch the usage. On Tue, Jun 28, 2011 at 4:18 PM, xin123620 <[hidden email]> wrote: > Thank you Jholtman. > Now count is 46001902. I was trying to retrieve one-year data, but I still > receive the following message: > "Error: cannot allocate vector of size 64.0 Mb" > > -- > View this message in context: http://r.789695.n4.nabble.com/Error-cannot-allocate-vector-of-size-tp3629384p3631354.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > [hidden email] mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? ______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. |
Thank you for replying. when I've tried to run the R syntax in a 64 bit computer,the problem is solved. Thank you for helping out. I totally agree your advice.
I would like to answer all your questions in case other people meet the same problem. The data contains one timestamp column with time zone, one integer column, and 12 Boolean columns. I tried to run one month sample, before >memory.limit() [1] 3583 > memory.size(max=F) [1] 156.82 > memory.size(max=T) [1] 241.06 After > memory.limit() [1] 3583 > memory.size(max=F) [1] 199.97 > memory.size(max=T) [1] 241.56 |
Free forum by Nabble | Edit this page |