R process killed when allocating too large matrix (Mac OS X)

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

R process killed when allocating too large matrix (Mac OS X)

Marius Hofert-4
Dear expeRts,

The following code leads to R being killed (under Mac OS X 10.11.4; R
installed from source; also happened under a previous unstable
version):

m <- matrix(0, 100000, 100000)

I expected an error that a vector of this size could not be allocated.

Besides the above (a bug?), how can I find out beforehand whether or
not a square matrix can be allocated?

Cheers,
Marius


PS: Here is the sessionInfo() output:

R version 3.3.0 (2016-05-03)
Platform: x86_64-apple-darwin15.4.0 (64-bit)
Running under: OS X 10.11.4 (El Capitan)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Simon Urbanek

On May 3, 2016, at 9:51 PM, Marius Hofert <[hidden email]> wrote:

> Dear expeRts,
>
> The following code leads to R being killed (under Mac OS X 10.11.4; R
> installed from source; also happened under a previous unstable
> version):
>
> m <- matrix(0, 100000, 100000)
>
> I expected an error that a vector of this size could not be allocated.
>
> Besides the above (a bug?), how can I find out beforehand whether or
> not a square matrix can be allocated?
>

Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).

Cheers,
Simon



> Cheers,
> Marius
>
>
> PS: Here is the sessionInfo() output:
>
> R version 3.3.0 (2016-05-03)
> Platform: x86_64-apple-darwin15.4.0 (64-bit)
> Running under: OS X 10.11.4 (El Capitan)
>
> locale:
> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Marius Hofert-4
In reply to this post by Marius Hofert-4
> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).

Hi Simon!

Sure, but who do you tell it if you don't know the killer?
This is all the killer left me with, the 'crime scene' if you like :-)

> m <- matrix(0, 90000, 100000)
Killed: 9

My colleague Wayne Oldford also tried it on his Mac machine and
apparently the killer went further down the hallway to his office
now... so scary. Here is Wayne's sessionInfo():

> sessionInfo()
R version 3.2.4 (2016-03-10)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.5 (Yosemite)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] rsconnect_0.4.1.11 tools_3.2.4
>

Cheers,
M

>
> Cheers,
> Simon

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Simon Urbanek

On May 4, 2016, at 6:14 PM, Marius Hofert <[hidden email]> wrote:

>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).
>
> Hi Simon!
>
> Sure, but who do you tell it if you don't know the killer?
> This is all the killer left me with, the 'crime scene' if you like :-)
>
>> m <- matrix(0, 90000, 100000)
> Killed: 9
>
> My colleague Wayne Oldford also tried it on his Mac machine and
> apparently the killer went further down the hallway to his office
> now... so scary. Here is Wayne's sessionInfo():
>

Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see

May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)

so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning.

Cheers,
Simon


>> sessionInfo()
> R version 3.2.4 (2016-03-10)
> Platform: x86_64-apple-darwin13.4.0 (64-bit)
> Running under: OS X 10.10.5 (Yosemite)
>
> locale:
> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> loaded via a namespace (and not attached):
> [1] rsconnect_0.4.1.11 tools_3.2.4
>>
>
> Cheers,
> M
>
>>
>> Cheers,
>> Simon
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Marius Hofert-4
In reply to this post by Marius Hofert-4
Hi Simon,

thanks for your quick reply.

1) ... so you can reproduce this?
2) Do you know a way how this can be 'foreseen'? We allocate larger
matrices in the copula package depending on the user's input
dimension. It would be good to tell her/him "Your dimension is quite
large. Be aware of killers in your neighborhood"... before the killer
attacks.

Thanks & cheers,
Marius


On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
<[hidden email]> wrote:

>
> On May 4, 2016, at 6:14 PM, Marius Hofert <[hidden email]> wrote:
>
>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).
>>
>> Hi Simon!
>>
>> Sure, but who do you tell it if you don't know the killer?
>> This is all the killer left me with, the 'crime scene' if you like :-)
>>
>>> m <- matrix(0, 90000, 100000)
>> Killed: 9
>>
>> My colleague Wayne Oldford also tried it on his Mac machine and
>> apparently the killer went further down the hallway to his office
>> now... so scary. Here is Wayne's sessionInfo():
>>
>
> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see
>
> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>
> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning.
>
> Cheers,
> Simon
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Simon Urbanek

On May 4, 2016, at 9:00 PM, Marius Hofert <[hidden email]> wrote:

> Hi Simon,
>
> thanks for your quick reply.
>
> 1) ... so you can reproduce this?

Yes, I can on 10.11.4.


> 2) Do you know a way how this can be 'foreseen'? We allocate larger
> matrices in the copula package depending on the user's input
> dimension. It would be good to tell her/him "Your dimension is quite
> large. Be aware of killers in your neighborhood"... before the killer
> attacks.
>

Not directly, because the system is happy to accommodate R's request for more memory but then it will strike out of the blue. Since that decision is made by the kernel and it doesn't expose its thinking about how it feels, it's hard to tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, unfortunately.

You could guess - e.g. by checking the total system memory

int mib [] = { CTL_HW, HW_MEMSIZE };
int64_t value = 0;
size_t length = sizeof(value);
sysctl(mib, 2, &value, &length, NULL, 0);

and comparing it to the sizes involved. However, even that is not foolproof, because it all depends on the other processes' memory usage, swap space size etc. There are slightly more involved ways to query the VM system as well, but I'm not sure I'd want to go so deep into the weeds, especially since this becomes quickly highly OS-specific.

Cheers,
Simon



> Thanks & cheers,
> Marius
>
>
> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
> <[hidden email]> wrote:
>>
>> On May 4, 2016, at 6:14 PM, Marius Hofert <[hidden email]> wrote:
>>
>>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).
>>>
>>> Hi Simon!
>>>
>>> Sure, but who do you tell it if you don't know the killer?
>>> This is all the killer left me with, the 'crime scene' if you like :-)
>>>
>>>> m <- matrix(0, 90000, 100000)
>>> Killed: 9
>>>
>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>> apparently the killer went further down the hallway to his office
>>> now... so scary. Here is Wayne's sessionInfo():
>>>
>>
>> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see
>>
>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>>
>> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning.
>>
>> Cheers,
>> Simon
>>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Hervé Pagès-2
Hi,

Interesting "feature" in 10.11.4. I wonder if the process is killed
before or after malloc() returns. If before, it seems very blunt:
"You're asking too much and I don't like it so I kill you now".
If after it doesn't look much better: "You're asking a lot and I
don't like it but I give it to you anyway. I'll kill you quickly
later".

Why wouldn't the kernel just refuse to give that memory instead
(i.e. malloc() returns NULL).

Just curious...

H.

On 05/04/2016 06:21 PM, Simon Urbanek wrote:

>
> On May 4, 2016, at 9:00 PM, Marius Hofert <[hidden email]> wrote:
>
>> Hi Simon,
>>
>> thanks for your quick reply.
>>
>> 1) ... so you can reproduce this?
>
> Yes, I can on 10.11.4.
>
>
>> 2) Do you know a way how this can be 'foreseen'? We allocate larger
>> matrices in the copula package depending on the user's input
>> dimension. It would be good to tell her/him "Your dimension is quite
>> large. Be aware of killers in your neighborhood"... before the killer
>> attacks.
>>
>
> Not directly, because the system is happy to accommodate R's request for more memory but then it will strike out of the blue. Since that decision is made by the kernel and it doesn't expose its thinking about how it feels, it's hard to tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, unfortunately.
>
> You could guess - e.g. by checking the total system memory
>
> int mib [] = { CTL_HW, HW_MEMSIZE };
> int64_t value = 0;
> size_t length = sizeof(value);
> sysctl(mib, 2, &value, &length, NULL, 0);
>
> and comparing it to the sizes involved. However, even that is not foolproof, because it all depends on the other processes' memory usage, swap space size etc. There are slightly more involved ways to query the VM system as well, but I'm not sure I'd want to go so deep into the weeds, especially since this becomes quickly highly OS-specific.
>
> Cheers,
> Simon
>
>
>
>> Thanks & cheers,
>> Marius
>>
>>
>> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
>> <[hidden email]> wrote:
>>>
>>> On May 4, 2016, at 6:14 PM, Marius Hofert <[hidden email]> wrote:
>>>
>>>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).
>>>>
>>>> Hi Simon!
>>>>
>>>> Sure, but who do you tell it if you don't know the killer?
>>>> This is all the killer left me with, the 'crime scene' if you like :-)
>>>>
>>>>> m <- matrix(0, 90000, 100000)
>>>> Killed: 9
>>>>
>>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>>> apparently the killer went further down the hallway to his office
>>>> now... so scary. Here is Wayne's sessionInfo():
>>>>
>>>
>>> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see
>>>
>>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>>>
>>> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning.
>>>
>>> Cheers,
>>> Simon
>>>
>>
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: [hidden email]
Phone:  (206) 667-5791
Fax:    (206) 667-1319

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Simon Urbanek

On May 4, 2016, at 9:49 PM, Hervé Pagès <[hidden email]> wrote:

> Hi,
>
> Interesting "feature" in 10.11.4. I wonder if the process is killed
> before or after malloc() returns. If before, it seems very blunt:
> "You're asking too much and I don't like it so I kill you now".
> If after it doesn't look much better: "You're asking a lot and I
> don't like it but I give it to you anyway. I'll kill you quickly
> later".
>
> Why wouldn't the kernel just refuse to give that memory instead
> (i.e. malloc() returns NULL).
>
> Just curious...
>

You can google it, but this seems to be standard practice in modern OSes (including OS X and Linux) where the kernel is not actually allocating the physical memory when the application asks for it so malloc() will succeed and give only a pointer to a virtual space where it will be mapping physical memory into as needed (aka opportunistic allocation). However, when the OS figures out later that it is overcommitted, it has no good way to signal the app that it can't use what it was promised, so it will use SIGKILL. The rationale is that in most cases the kernel can scramble to fulfill the request so it gives it more flexibility in managing the memory (it can spread pages to RAM, swap, shared ,...), but when it cannot recover it has no choice but to kill the process.

I don't have the debugger open anymore, but if I recall correctly in my case the kill happened in copyVector() -> duplicate(), so it was not necessarily on the initial allocation but rather further down the line.

Cheers,
Simon



> H.
>
> On 05/04/2016 06:21 PM, Simon Urbanek wrote:
>>
>> On May 4, 2016, at 9:00 PM, Marius Hofert <[hidden email]> wrote:
>>
>>> Hi Simon,
>>>
>>> thanks for your quick reply.
>>>
>>> 1) ... so you can reproduce this?
>>
>> Yes, I can on 10.11.4.
>>
>>
>>> 2) Do you know a way how this can be 'foreseen'? We allocate larger
>>> matrices in the copula package depending on the user's input
>>> dimension. It would be good to tell her/him "Your dimension is quite
>>> large. Be aware of killers in your neighborhood"... before the killer
>>> attacks.
>>>
>>
>> Not directly, because the system is happy to accommodate R's request for more memory but then it will strike out of the blue. Since that decision is made by the kernel and it doesn't expose its thinking about how it feels, it's hard to tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, unfortunately.
>>
>> You could guess - e.g. by checking the total system memory
>>
>> int mib [] = { CTL_HW, HW_MEMSIZE };
>> int64_t value = 0;
>> size_t length = sizeof(value);
>> sysctl(mib, 2, &value, &length, NULL, 0);
>>
>> and comparing it to the sizes involved. However, even that is not foolproof, because it all depends on the other processes' memory usage, swap space size etc. There are slightly more involved ways to query the VM system as well, but I'm not sure I'd want to go so deep into the weeds, especially since this becomes quickly highly OS-specific.
>>
>> Cheers,
>> Simon
>>
>>
>>
>>> Thanks & cheers,
>>> Marius
>>>
>>>
>>> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
>>> <[hidden email]> wrote:
>>>>
>>>> On May 4, 2016, at 6:14 PM, Marius Hofert <[hidden email]> wrote:
>>>>
>>>>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :).
>>>>>
>>>>> Hi Simon!
>>>>>
>>>>> Sure, but who do you tell it if you don't know the killer?
>>>>> This is all the killer left me with, the 'crime scene' if you like :-)
>>>>>
>>>>>> m <- matrix(0, 90000, 100000)
>>>>> Killed: 9
>>>>>
>>>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>>>> apparently the killer went further down the hallway to his office
>>>>> now... so scary. Here is Wayne's sessionInfo():
>>>>>
>>>>
>>>> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see
>>>>
>>>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>>>>
>>>> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning.
>>>>
>>>> Cheers,
>>>> Simon
>>>>
>>>
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: [hidden email]
> Phone:  (206) 667-5791
> Fax:    (206) 667-1319
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Marius Hofert-4
In reply to this post by Hervé Pagès-2
Hi Simon,

... all interesting (but quite a bit above my head). I only read
'Linux' and want to throw in that this problem does not appear on
Linux (it seems). I talked about this with Martin Maechler and he
reported that the same example (on one of his machines; with NA_real_
instead of '0's in the matrix) gave:

  Error: cannot allocate vector of size 70.8 Gb
    Timing stopped at: 144.79 41.619 202.019

... but no killer around...

Cheers,
Marius

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Uwe Ligges-3


On 05.05.2016 04:25, Marius Hofert wrote:

> Hi Simon,
>
> ... all interesting (but quite a bit above my head). I only read
> 'Linux' and want to throw in that this problem does not appear on
> Linux (it seems). I talked about this with Martin Maechler and he
> reported that the same example (on one of his machines; with NA_real_
> instead of '0's in the matrix) gave:
>
>   Error: cannot allocate vector of size 70.8 Gb
>     Timing stopped at: 144.79 41.619 202.019
>
> ... but no killer around...

Well, with n=1. ;-)

Actually this also happens under Linux and I had my R processes killed
more than once (and much worse also other processes so that we had to
reboot a server, essentially). That's why we use job scheduling on
servers for R nowadays ...

Best,
Uwe

>
> Cheers,
> Marius
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Prof Brian Ripley
On 05/05/2016 10:11, Uwe Ligges wrote:

>
>
> On 05.05.2016 04:25, Marius Hofert wrote:
>> Hi Simon,
>>
>> ... all interesting (but quite a bit above my head). I only read
>> 'Linux' and want to throw in that this problem does not appear on
>> Linux (it seems). I talked about this with Martin Maechler and he
>> reported that the same example (on one of his machines; with NA_real_
>> instead of '0's in the matrix) gave:
>>
>>   Error: cannot allocate vector of size 70.8 Gb
>>     Timing stopped at: 144.79 41.619 202.019
>>
>> ... but no killer around...
>
> Well, with n=1. ;-)
>
> Actually this also happens under Linux and I had my R processes killed
> more than once (and much worse also other processes so that we had to
> reboot a server, essentially). That's why we use job scheduling on
> servers for R nowadays ...

Yes, Linux does not deal safely with running out of memory, although it
is better than it was.  In my experience, only commercial Unices do that
gracefully.

Have you tried setting a (virtual) memory limit on the process using the
shell it is launched from?  I have found that to be effective on most
OSes, at least in protecting other processes from being killed.
However, some things do reserve excessive amounts of VM that they do not
use and so cannot be run under a sensible limit.

--
Brian D. Ripley,                  [hidden email]
Emeritus Professor of Applied Statistics, University of Oxford

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Jeroen Ooms
On 05/05/2016 10:11, Uwe Ligges wrote:
> Actually this also happens under Linux and I had my R processes killed
> more than once (and much worse also other processes so that we had to
> reboot a server, essentially).

I found that setting RLIMIT_AS [1] works very well on Linux. But this
requires that you cap memory to some fixed value.

> library(RAppArmor)
> rlimit_as(1e9)
> rnorm(1e9)
Error: cannot allocate vector of size 7.5 Gb

The RAppArmor package has many other utilities to protect your server
such from a mis-behaving process such as limiting cpu time
(RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).

[1] http://linux.die.net/man/2/getrlimit

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Kirill Müller
My ulimit package exposes this API ([1], should finally submit it to
CRAN); unfortunately this very API seems to be unsupported on OS X
[2,3]. Last time I looked into it, neither of the documented settings
achieved the desired effect.


-Kirill


[1] http://krlmlr.github.io/ulimit
[2]
http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
[3]
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


On 10.05.2016 01:08, Jeroen Ooms wrote:

> On 05/05/2016 10:11, Uwe Ligges wrote:
>> Actually this also happens under Linux and I had my R processes killed
>> more than once (and much worse also other processes so that we had to
>> reboot a server, essentially).
> I found that setting RLIMIT_AS [1] works very well on Linux. But this
> requires that you cap memory to some fixed value.
>
>> library(RAppArmor)
>> rlimit_as(1e9)
>> rnorm(1e9)
> Error: cannot allocate vector of size 7.5 Gb
>
> The RAppArmor package has many other utilities to protect your server
> such from a mis-behaving process such as limiting cpu time
> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).
>
> [1] http://linux.die.net/man/2/getrlimit
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Martin Maechler
>>>>> Kirill Müller <[hidden email]>
>>>>>     on Wed, 11 May 2016 10:42:56 +0200 writes:

    > My ulimit package exposes this API ([1], should finally submit it to
    > CRAN); unfortunately this very API seems to be unsupported on OS X
    > [2,3]. Last time I looked into it, neither of the documented settings
    > achieved the desired effect.

    > -Kirill

    > [1] http://krlmlr.github.io/ulimit
    > [2]
    > http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
    > [3]
    > https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


    > On 10.05.2016 01:08, Jeroen Ooms wrote:
    >> On 05/05/2016 10:11, Uwe Ligges wrote:
    >>> Actually this also happens under Linux and I had my R processes killed
    >>> more than once (and much worse also other processes so that we had to
    >>> reboot a server, essentially).

I agree that Linux is not consistently fine here either.

    >> I found that setting RLIMIT_AS [1] works very well on Linux. But this
    >> requires that you cap memory to some fixed value.

conceivably one could set a default cap, using something
equivalent to the data in

                sfsmisc::Sys.meminfo()
or sfsmisc::Sys.memGB()

(very simple, > 10 year old interfaces, based on the  Linux-only (?)
 '/proc/*' filesystem).


In an ideal word, some of us,
    from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
    maintainer("microbenchmark>, ...
would sit together and devise an R function interface (based on
low level platform specific interfaces, specifically for at least
Linux/POSIX-compliant, Mac, and Windows) which would allow
something  like your rlimit(..) calls below.

We'd really need something to work on all platforms ideally,
to be used by R package maintainers
and possibly even better by R itself at startup, setting a
reasonable memory cap - which the user could raise even to +Inf (or lower
even more).

Martin

---------
Final notes about RAppArmor, not relevant to the main thread topic :

Note: I'm working in pretty well maintained Fedora Linux env,
but Apparmor is not only not activated, but even not available.
OTOH, usting  RLIMIT / gerlimit on Linux is very generally
available.

As a consequence, the three last lines of

   > require(RAppArmor)
   Loading required package: RAppArmor
   Loading required package: tools
   Failed to lookup process confinement:
   AppArmor not supported on this system
   Have a look at: sudo aa-status

were very confusing to me:  My conclusion was I could not use
the RAppArmor package.

(But that's wrong: For the  rlimit*()  functions below, one do
 *NOT* need an AppArmor-enabled version of Linux !)
                 
    >>> library(RAppArmor)
    >>> rlimit_as(1e9)
    >>> rnorm(1e9)
    >> Error: cannot allocate vector of size 7.5 Gb
    >>
    >> The RAppArmor package has many other utilities to protect your server
    >> such from a mis-behaving process such as limiting cpu time
    >> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).
    >>
    >> [1] http://linux.die.net/man/2/getrlimit

and from my current explorations I gather that all of these are
*not* Apparmor related... so could/should maybe rather migrate
into a lightweight package not mentioning AppArmor ?

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Jeroen Ooms.
On Thu, May 12, 2016 at 9:51 AM, Martin Maechler
<[hidden email]> wrote:
> My conclusion was I could not use the RAppArmor package.
>
> (But that's wrong: For the  rlimit*()  functions below, one do
>  *NOT* need an AppArmor-enabled version of Linux !)

Yes, it is a relatively recent (unadvertised) feature that the package
now builds on linux systems without libapparmor. I agree this names
the package name confusing. I'll make at least that warning more
informative.

Some background: When I started the package (5 years ago) I expected
that soon all linux distributions would have the apparmor module which
has been in the kernel since 2.6.36. However Redhat is explicitly
disabling it because they are pushing a competing MAC system (SELinux)
which they develop together with the NSA, and they really want you to
use this instead (and only this!).

> I gather that all of these are *not* Apparmor related... so could/should maybe rather migrate into a lightweight package not mentioning AppArmor ?

I agree, it has been on the to do list for a while; Kirill and me were
talking yesterday about what would be the best route to take:

 - A small package with only the rlimit bindings
 - or: A 'linux' package with bindings to anything in the kernel,
including rlimit, but possibly other system tools.
 - or: A package targeting POSIX/unix with standard functionality that
is also available on OSX/BSD.

>From my experience, windows is pretty useless for this kind of stuff.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

virgo
On Thu, 12 May 2016 12:07:29 +0200
Jeroen Ooms <[hidden email]> wrote:

> On Thu, May 12, 2016 at 9:51 AM, Martin Maechler
> <[hidden email]> wrote:
> > My conclusion was I could not use the RAppArmor package.
> >
> > (But that's wrong: For the  rlimit*()  functions below, one do
> >  *NOT* need an AppArmor-enabled version of Linux !)  
>
> Yes, it is a relatively recent (unadvertised) feature that the package
> now builds on linux systems without libapparmor. I agree this names
> the package name confusing. I'll make at least that warning more
> informative.
>
> Some background: When I started the package (5 years ago) I expected
> that soon all linux distributions would have the apparmor module which
> has been in the kernel since 2.6.36. However Redhat is explicitly
> disabling it because they are pushing a competing MAC system (SELinux)
> which they develop together with the NSA, and they really want you to
> use this instead (and only this!).
>
> > I gather that all of these are *not* Apparmor related... so
> > could/should maybe rather migrate into a lightweight package not
> > mentioning AppArmor ?  
>
> I agree, it has been on the to do list for a while; Kirill and me were
> talking yesterday about what would be the best route to take:
>
>  - A small package with only the rlimit bindings
>  - or: A 'linux' package with bindings to anything in the kernel,
> including rlimit, but possibly other system tools.
>  - or: A package targeting POSIX/unix with standard functionality that
> is also available on OSX/BSD.
>
> >From my experience, windows is pretty useless for this kind of
> >stuff.  
Maybe not so useless after reading [1] about computationally
querying the OSes on the available memory and [2] about pushing the
OSes' limits. The latter page is part of a series where each topic is
valuable on its own.

[1] https://msdn.microsoft.com/en-us/library/aa366778.aspx
[2] https://blogs.technet.microsoft.com/markrussinovich/2008/07/21/pushing-the-limits-of-windows-physical-memory/
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R process killed when allocating too large matrix (Mac OS X)

Kirill Müller
In reply to this post by Martin Maechler
On 12.05.2016 09:51, Martin Maechler wrote:

>      > My ulimit package exposes this API ([1], should finally submit it to
>      > CRAN); unfortunately this very API seems to be unsupported on OS X
>      > [2,3]. Last time I looked into it, neither of the documented settings
>      > achieved the desired effect.
>
>      > -Kirill
>
>      > [1] http://krlmlr.github.io/ulimit
>      > [2]
>      > http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
>      > [3]
>      > https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html
>
>
> ...
>
> In an ideal word, some of us,
>      from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
>      maintainer("microbenchmark>, ...
> would sit together and devise an R function interface (based on
> low level platform specific interfaces, specifically for at least
> Linux/POSIX-compliant, Mac, and Windows) which would allow
> something  like your rlimit(..) calls below.
>
> We'd really need something to work on all platforms ideally,
> to be used by R package maintainers
> and possibly even better by R itself at startup, setting a
> reasonable memory cap - which the user could raise even to +Inf (or lower
> even more).
>
I haven't found a Windows API that allows limiting the address space,
only one that limits the working set size; it seems likely that this is
the best we can get on OS X, too, but then my experience with OS X is
very limited.

mallinfo() is used on Windows and seems to be available on Linux, too,
but not on OS X.


-Kirill

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel