Quantcast

how to read CSV file in R?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

how to read CSV file in R?

dhanush
I tried to read a CSV file in R. The file has about 100,000 records and 75 columns. When used read.delim, I got this error. I am using R ver 10.1.

> los<-read.delim("file.csv",header=T,sep=",")
Warning message:
In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
  Reached total allocation of 1535Mb: see help(memory.size)

Thanks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to read CSV file in R?

Joris FA Meys
That will be R 2.10.1 if I'm correct.

For reading in csv files, there's a function read.csv who does just that:
los <- read.csv("file.csv",header=T)

But that is just a detail. You have problems with your memory, but
that's not caused by the size of your dataframe. On my system, a
matrix with 100,000 rows and 75 columns takes only 28 Mb. So I guess
your workspace is cluttered with other stuff.

Check following help pages :
?Memory
?memory.size
?Memory.limits

it generally doesn't make a difference, but sometimes using gc() can
set some memory free again.

If none of this information helps, please provide us with a bit more
info regarding your system and the content of your current workspace.

Cheers
Joris

On Tue, Jun 8, 2010 at 8:46 AM, dhanush <[hidden email]> wrote:

>
> I tried to read a CSV file in R. The file has about 100,000 records and 75
> columns. When used read.delim, I got this error. I am using R ver 10.1.
>
>> los<-read.delim("file.csv",header=T,sep=",")
> Warning message:
> In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
>  Reached total allocation of 1535Mb: see help(memory.size)
>
> Thanks
> --
> View this message in context: http://r.789695.n4.nabble.com/how-to-read-CSV-file-in-R-tp2246930p2246930.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>



--
Joris Meys
Statistical consultant

Ghent University
Faculty of Bioscience Engineering
Department of Applied mathematics, biometrics and process control

tel : +32 9 264 59 87
[hidden email]
-------------------------------
Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: how to read CSV file in R?

Jens Oehlschlägel-2
If you have memory problems reading csv you can use read.csv.ffdf from package ff which reads in chunks.
The result is a ffdf object, say myffdf, from which binary subscripting [,] returns standard data.frames, such as
myffdf[,]  # returns all data (if it fits into memory)
myffdf[somerows,] # returns a subset of data

Do read and understand the help concerning filename location and implications for finalizers and permanency.

Cheers
Jens Oehlschlägel


-----Ursprüngliche Nachricht-----
Von: Joris Meys <[hidden email]>
Gesendet: Jun 8, 2010 1:11:20 PM
An: dhanush <[hidden email]>
Betreff: Re: [R] how to read CSV file in R?

>That will be R 2.10.1 if I'm correct.
>
>For reading in csv files, there's a function read.csv who does just that:
>los <- read.csv("file.csv",header=T)
>
>But that is just a detail. You have problems with your memory, but
>that's not caused by the size of your dataframe. On my system, a
>matrix with 100,000 rows and 75 columns takes only 28 Mb. So I guess
>your workspace is cluttered with other stuff.
>
>Check following help pages :
>?Memory
>?memory.size
>?Memory.limits
>
>it generally doesn't make a difference, but sometimes using gc() can
>set some memory free again.
>
>If none of this information helps, please provide us with a bit more
>info regarding your system and the content of your current workspace.
>
>Cheers
>Joris
>
>On Tue, Jun 8, 2010 at 8:46 AM, dhanush <[hidden email]> wrote:
>>
>> I tried to read a CSV file in R. The file has about 100,000 records and 75
>> columns. When used read.delim, I got this error. I am using R ver 10.1.
>>
>>> los<-read.delim("file.csv",header=T,sep=",")
>> Warning message:
>> In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
>>  Reached total allocation of 1535Mb: see help(memory.size)
>>
>> Thanks
>> --
>> View this message in context: http://r.789695.n4.nabble.com/how-to-read-CSV-file-in-R-tp2246930p2246930.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>
>
>--
>Joris Meys
>Statistical consultant
>
>Ghent University
>Faculty of Bioscience Engineering
>Department of Applied mathematics, biometrics and process control
>
>tel : +32 9 264 59 87
>[hidden email]
>-------------------------------
>Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php
>
>______________________________________________
>[hidden email] mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Loading...