memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

Ken Termiso
Hi all,

Sorry if this is a dumb question, but I am on 10.4 with R2.2, and when
loading a big text file (~500MB) with scan(file, what=character) I am
throwing malloc errors that say I am out of memory...I have 5GB on this
machine, and Activity Monitor tells me R is only up to ~1.84GB both times
this has happened (running from terminal)...

I am wondering why this is happening when I still have >2GB of free memory
waiting to be used...?

Any advice would be much obliged,
Ken

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

David Ruau
Hi,
I don't know why, but I have a workaround maybe:
You can load sequentially the file. Split the text file in 2 or 3 and
re-associate the vector/list into r after.
Once I was using a similar technic to write a huge matrix into a txt
file.

David

On Dec 14, 2005, at 21:47, Ken Termiso wrote:

> Hi all,
>
> Sorry if this is a dumb question, but I am on 10.4 with R2.2, and when
> loading a big text file (~500MB) with scan(file, what=character) I am
> throwing malloc errors that say I am out of memory...I have 5GB on this
> machine, and Activity Monitor tells me R is only up to ~1.84GB both
> times
> this has happened (running from terminal)...
>
> I am wondering why this is happening when I still have >2GB of free
> memory
> waiting to be used...?
>
> Any advice would be much obliged,
> Ken
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

Roger D. Peng
In reply to this post by Ken Termiso
I'm not completely sure, but I don't think OS X is at the point yet where it can
access > 2GB of memory (like, for example, Linux on Opteron).  More
specifically, I'm not sure a single process image can access > 2GB of memory,
but I'd welcome any corrections to that statement.  To be sure, this problem is
not an issue with R because R has regularly been reported to access u> 4GB of
memory when the OS allows it.

-roger

Ken Termiso wrote:

> Hi all,
>
> Sorry if this is a dumb question, but I am on 10.4 with R2.2, and when
> loading a big text file (~500MB) with scan(file, what=character) I am
> throwing malloc errors that say I am out of memory...I have 5GB on this
> machine, and Activity Monitor tells me R is only up to ~1.84GB both times
> this has happened (running from terminal)...
>
> I am wondering why this is happening when I still have >2GB of free memory
> waiting to be used...?
>
> Any advice would be much obliged,
> Ken
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>

--
Roger D. Peng  |  http://www.biostat.jhsph.edu/~rpeng/

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

Albert Vilella
El dj 15 de 12 del 2005 a les 08:17 -0500, en/na Roger D. Peng va
escriure:
> I'm not completely sure, but I don't think OS X is at the point yet where it can
> access > 2GB of memory (like, for example, Linux on Opteron).  More
> specifically, I'm not sure a single process image can access > 2GB of memory,
> but I'd welcome any corrections to that statement.  To be sure, this problem is
> not an issue with R because R has regularly been reported to access u> 4GB of
> memory when the OS allows it.

I may seen somewhere that OSX has a "per process" limit of ~1.5GB max
RAM,

I think that WinXP has a limit on 2GB RAM,

In Linux, I believe a process can take ~4GB, more on x86-64 platforms
either directly or tweaking some option in the kernel,

Not sure though, I'm pretty sure someone else can give a clearer picture
on this,

    Albert.

>
> -roger
>
> Ken Termiso wrote:
> > Hi all,
> >
> > Sorry if this is a dumb question, but I am on 10.4 with R2.2, and when
> > loading a big text file (~500MB) with scan(file, what=character) I am
> > throwing malloc errors that say I am out of memory...I have 5GB on this
> > machine, and Activity Monitor tells me R is only up to ~1.84GB both times
> > this has happened (running from terminal)...
> >
> > I am wondering why this is happening when I still have >2GB of free memory
> > waiting to be used...?
> >
> > Any advice would be much obliged,
> > Ken
> >
> > ______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
> >
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

Sean Davis



On 12/15/05 10:36 AM, "Albert Vilella" <[hidden email]> wrote:

> El dj 15 de 12 del 2005 a les 08:17 -0500, en/na Roger D. Peng va
> escriure:
>> I'm not completely sure, but I don't think OS X is at the point yet where it
>> can
>> access > 2GB of memory (like, for example, Linux on Opteron).  More
>> specifically, I'm not sure a single process image can access > 2GB of memory,
>> but I'd welcome any corrections to that statement.  To be sure, this problem
>> is
>> not an issue with R because R has regularly been reported to access u> 4GB of
>> memory when the OS allows it.
>
> I may seen somewhere that OSX has a "per process" limit of ~1.5GB max
> RAM,

I don't think this is true.  See here for Apple's answer:

http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemo
ry/index.html

Note that receiving a malloc error means that malloc failed.  Therefore,
what you see in ActivityMonitor is the memory consumption BEFORE a large
memory block is allocated, if I understand things correctly.  Correct me if
I'm wrong here, please.

Sean

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: memory tops out at 1.84gb on OS X 10.4 machine w/ 5GB ram

Prof Brian Ripley
The issue here is not the per-process addess space, which is 4Gb on all
32-bit OSes (more or less by definition), but the user address space of
the process.  That will be less, and how much less depends on the OS.
For most it seems to be 3Gb for the user, 1Gb for the system, but Windows
by default has 2Gb for each.  I've never seen an authorative figure for
MacOS X.

That's one issue.  Another is that with a 3Gb address space, it is likely
that there will be no large contiguous blocks once more than half of it is
in use, so memory allocation can fail even if there is memory available
(as if is fragmented).

See ?"Memory-limits" and the R-admin manual (on 32- vs 64-bit versions)
for further discussion.

Note that although Apple claims to have a 64-bit OS, the reality is
that they have made some small steps in that direction:

http://arstechnica.com/reviews/os/macosx-10.4.ars/4

AFAIK the precompiled version of R is 32-bit only rather than the 'fat
binary' discussed there, and in any case GUI applications are restricted
to 32-bit.

Note that we don't even know if 'Jerk Alert' was talking about MacOS, as
he did not actually say so.  But only MacOS users seems to refer to a
operating system using two Tens (one Roman).


On Thu, 15 Dec 2005, Sean Davis wrote:

> On 12/15/05 10:36 AM, "Albert Vilella" <[hidden email]> wrote:
>
>> El dj 15 de 12 del 2005 a les 08:17 -0500, en/na Roger D. Peng va
>> escriure:
>>> I'm not completely sure, but I don't think OS X is at the point yet where it
>>> can
>>> access > 2GB of memory (like, for example, Linux on Opteron).  More
>>> specifically, I'm not sure a single process image can access > 2GB of memory,
>>> but I'd welcome any corrections to that statement.  To be sure, this problem
>>> is
>>> not an issue with R because R has regularly been reported to access u> 4GB of
>>> memory when the OS allows it.
>>
>> I may seen somewhere that OSX has a "per process" limit of ~1.5GB max
>> RAM,
>
> I don't think this is true.  See here for Apple's answer:
>
> http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemo
> ry/index.html
>
> Note that receiving a malloc error means that malloc failed.  Therefore,
> what you see in ActivityMonitor is the memory consumption BEFORE a large
> memory block is allocated, if I understand things correctly.  Correct me if
> I'm wrong here, please.


--
Brian D. Ripley,                  [hidden email]
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Loading...