mmap error-12, macbook pro

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

mmap error-12, macbook pro

Mamun
Dear List,

I am trying to read some files using read.csv and total size of those files are 3.99 GB. I am using MacBook Pro with 4GB RAM(snow leopard). I also tried to run a chunk from those files and altogether the size was 1.33 GB. But every time I was getting the following error

R(1200) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

I managed to run smaller chunks (400MB) and I also saved the object. But problem is when I try to load some of those objects(which is not more than 1.5 GB altogether), I get same error again. Can someone please help? why am I getting this error? does R need more space than the actual file size? I may buy new machine if its something related to RAM size. but if it is some R problem then it will be very useless to buy new machine. So, I really need to know why this is happening.

Any help is appreciated.

Thanks in advance.

regards,
Mamun
Reply | Threaded
Open this post in threaded view
|

Re: mmap error-12, macbook pro

Thomas Lumley


There seem to have been three almost identical reports of this from three different email addresses, including two widely-separate UK academic institutions, and two user names, which is a bit weird.

However:  R  will need more than 4GB memory to handle a 3.99GB file, and probably more than 4GB memory to handle a 1.33GB file.

On 32-bit R I think the recommendation is that the data should occupy no more than 10% of the address space, ie, about 400MB.  For 64-bit R, you are likely to still find R very slow if the data set is more than about 1/3 of physical memory.   For a 4GB data file I would recommend a computer with at least 16GB memory.

Now, it is often possible to load only a small fraction of the data at one time and so to analyse large data sets on smaller computers.  Three examples that I have worked on
     - analysing data from a survey data set about the same size as your data, by keeping most of the data in a SQLite database and just loading a
        few variables at a time.
     - fitting linear regression models to large data sets with the biglm package, by keeping the data in a SQLite database and just loading a few rows
        at a time
     - analysing whole-genome genetic data (40GB or so) by storing the data in a netCDF file and reading appropriate chunks with the ncdf package.

           -thomas


On Wed, 17 Mar 2010, Mamun wrote:

>
> Dear List,
>
> I am trying to read some files using read.csv and total size of those files
> are 3.99 GB. I am using MacBook Pro with 4GB RAM(snow leopard). I also tried
> to run a chunk from those files and altogether the size was 1.33 GB. But
> every time I was getting the following error
>
> R(1200) malloc: *** mmap(size=16777216) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
>
> I managed to run smaller chunks (400MB) and I also saved the object. But
> problem is when I try to load some of those objects(which is not more than
> 1.5 GB altogether), I get same error again. Can someone please help? why am
> I getting this error? does R need more space than the actual file size? I
> may buy new machine if its something related to RAM size. but if it is some
> R problem then it will be very useless to buy new machine. So, I really need
> to know why this is happening.
>
> Any help is appreciated.
>
> Thanks in advance.
>
> regards,
> Mamun
> --
> View this message in context: http://n4.nabble.com/mmap-error-12-macbook-pro-tp1596234p1596234.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

Thomas Lumley Assoc. Professor, Biostatistics
[hidden email] University of Washington, Seattle

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.