Help to create bugzilla account

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Help to create bugzilla account

Dmitriy Selivanov
Hi mailing list and R-core. Could someone from R-core please help me to
create account in bugzilla? I would like to submit issue related to gc() to
wishlist.

Related context is here -
https://stat.ethz.ch/pipermail/r-devel/2017-July/074715.html


--
Regards
Dmitriy Selivanov

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Martin Maechler
>>>>> Dmitriy Selivanov <[hidden email]>
>>>>>     on Fri, 11 Aug 2017 17:33:31 +0400 writes:

    > Hi mailing list and R-core. Could someone from R-core please help me to
    > create account in bugzilla? I would like to submit issue related to gc() to
    > wishlist.

I will create one.

Your previous e-mails left me pretty clueless about what the
problem is that you want to solve ... but maybe others
understand better what you mean.

Note that in the case of such a relatively sophisticated wish
without a clear sign of a problem (in my view)
chances are not high that anything will change, unless someone
provides a (small footprint) patch towards the (R-devel aka
"trunk") sources *and* reproducible R code that depicts the
problem.

Still: Thank you for trying to make R better by contributing with
careful bug reports !

Best,
Martin


    > Related context is here -
    > https://stat.ethz.ch/pipermail/r-devel/2017-July/074715.html


    > --
    > Regards
    > Dmitriy Selivanov

    > [[alternative HTML version deleted]]

    > ______________________________________________
    > [hidden email] mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Dmitriy Selivanov
Thanks Martin, I've received invitation and will create ticket soon.

Regarding issue - basically the problem is that on operating systems which
use glibc memory is not freed (R releases it, but system doesn't trim it).
Setting corresponding environment variable (MALLOC_TRIM_THRESHOLD_) doesn't
help, but manual call to `*malloc_trim*` does.
Implication is following - for long-running jobs linux OOM killer can kill
R process because highly-fragmented pieces of memory never returned back to
OS. Very simple example:

   1. I have 2gb of RAM on the machine.
   2. I create large list of small objects which occupies 1.5 gig.
   3. I remove it (even can manually call gc()) - "top" still shows 1.5 gig
   occupied. If I decide to recreate similar 1.5gig list - ram will be reused.
   4. However if after removal of the list I will decide to create normal
   continuous  integer/double vector of size 500+ mb - R will report that it
   can't allocate vector of such size.

Hope this helped.

2017-08-11 18:00 GMT+04:00 Martin Maechler <[hidden email]>:

> >>>>> Dmitriy Selivanov <[hidden email]>
> >>>>>     on Fri, 11 Aug 2017 17:33:31 +0400 writes:
>
>     > Hi mailing list and R-core. Could someone from R-core please help me
> to
>     > create account in bugzilla? I would like to submit issue related to
> gc() to
>     > wishlist.
>
> I will create one.
>
> Your previous e-mails left me pretty clueless about what the
> problem is that you want to solve ... but maybe others
> understand better what you mean.
>
> Note that in the case of such a relatively sophisticated wish
> without a clear sign of a problem (in my view)
> chances are not high that anything will change, unless someone
> provides a (small footprint) patch towards the (R-devel aka
> "trunk") sources *and* reproducible R code that depicts the
> problem.
>
> Still: Thank you for trying to make R better by contributing with
> careful bug reports !
>
> Best,
> Martin
>
>
>     > Related context is here -
>     > https://stat.ethz.ch/pipermail/r-devel/2017-July/074715.html
>
>
>     > --
>     > Regards
>     > Dmitriy Selivanov
>
>     > [[alternative HTML version deleted]]
>
>     > ______________________________________________
>     > [hidden email] mailing list
>     > https://stat.ethz.ch/mailman/listinfo/r-devel
>



--
Regards
Dmitriy Selivanov

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Iñaki Úcar
In reply to this post by Martin Maechler
2017-08-11 16:00 GMT+02:00 Martin Maechler <[hidden email]>:

>>>>>> Dmitriy Selivanov <[hidden email]>
>>>>>>     on Fri, 11 Aug 2017 17:33:31 +0400 writes:
>
>     > Hi mailing list and R-core. Could someone from R-core please help me to
>     > create account in bugzilla? I would like to submit issue related to gc() to
>     > wishlist.
>
> I will create one.
>
> Your previous e-mails left me pretty clueless about what the
> problem is that you want to solve ... but maybe others
> understand better what you mean.
>
> Note that in the case of such a relatively sophisticated wish
> without a clear sign of a problem (in my view)
> chances are not high that anything will change, unless someone
> provides a (small footprint) patch towards the (R-devel aka
> "trunk") sources *and* reproducible R code that depicts the
> problem.

How to reproduce it:

a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory
rm(a)
gc() # the R process still has the memory assigned

Iñaki

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Simon Urbanek

> On Aug 11, 2017, at 12:57 PM, Iñaki Úcar <[hidden email]> wrote:
>
> 2017-08-11 16:00 GMT+02:00 Martin Maechler <[hidden email]>:
>>>>>>> Dmitriy Selivanov <[hidden email]>
>>>>>>>    on Fri, 11 Aug 2017 17:33:31 +0400 writes:
>>
>>> Hi mailing list and R-core. Could someone from R-core please help me to
>>> create account in bugzilla? I would like to submit issue related to gc() to
>>> wishlist.
>>
>> I will create one.
>>
>> Your previous e-mails left me pretty clueless about what the
>> problem is that you want to solve ... but maybe others
>> understand better what you mean.
>>
>> Note that in the case of such a relatively sophisticated wish
>> without a clear sign of a problem (in my view)
>> chances are not high that anything will change, unless someone
>> provides a (small footprint) patch towards the (R-devel aka
>> "trunk") sources *and* reproducible R code that depicts the
>> problem.
>
> How to reproduce it:
>
> a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory
> rm(a)
> gc() # the R process still has the memory assigned
>

Right, but that's unavoidable because of the way Linux allocates memory - see FAQ 7.42
The memory is free, Linux just keeps it for future allocations.

Running malloc.trim doesn't help, because the issue is fragmentation due to the linear design of the pool - you likely will have another object on top so in most practical cases malloc.trim() doesn't actually do anything. You can always call malloc.trim() yourself is you think it helps, but it doesn't in the general case. The only way to address that would be to move allocated objects from top of the pool down, but that's not something R can allow, because it cannot know which code still has SEXP pointers referring to that object.

Cheers,
Simon

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Dmitriy Selivanov
Strange because in my all my experiments calling malloc.trim always helped
- memory reported by top decreased to the level it supposed to be. Do you
have in mind case when calling malloc.trim won't do anything? Also
shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim
calls? At the moment seems any value is ignored...

12 авг. 2017 г. 6:09 ПП пользователь "Simon Urbanek" <
[hidden email]> написал:

>
> > On Aug 11, 2017, at 12:57 PM, Iñaki Úcar <[hidden email]> wrote:
> >
> > 2017-08-11 16:00 GMT+02:00 Martin Maechler <[hidden email]>:
> >>>>>>> Dmitriy Selivanov <[hidden email]>
> >>>>>>>    on Fri, 11 Aug 2017 17:33:31 +0400 writes:
> >>
> >>> Hi mailing list and R-core. Could someone from R-core please help me to
> >>> create account in bugzilla? I would like to submit issue related to
> gc() to
> >>> wishlist.
> >>
> >> I will create one.
> >>
> >> Your previous e-mails left me pretty clueless about what the
> >> problem is that you want to solve ... but maybe others
> >> understand better what you mean.
> >>
> >> Note that in the case of such a relatively sophisticated wish
> >> without a clear sign of a problem (in my view)
> >> chances are not high that anything will change, unless someone
> >> provides a (small footprint) patch towards the (R-devel aka
> >> "trunk") sources *and* reproducible R code that depicts the
> >> problem.
> >
> > How to reproduce it:
> >
> > a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory
> > rm(a)
> > gc() # the R process still has the memory assigned
> >
>
> Right, but that's unavoidable because of the way Linux allocates memory -
> see FAQ 7.42
> The memory is free, Linux just keeps it for future allocations.
>
> Running malloc.trim doesn't help, because the issue is fragmentation due
> to the linear design of the pool - you likely will have another object on
> top so in most practical cases malloc.trim() doesn't actually do anything.
> You can always call malloc.trim() yourself is you think it helps, but it
> doesn't in the general case. The only way to address that would be to move
> allocated objects from top of the pool down, but that's not something R can
> allow, because it cannot know which code still has SEXP pointers referring
> to that object.
>
> Cheers,
> Simon
>
>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

luke-tierney
On Sat, 12 Aug 2017, Dmitriy Selivanov wrote:

> Strange because in my all my experiments calling malloc.trim always helped
> - memory reported by top decreased to the level it supposed to be. Do you
> have in mind case when calling malloc.trim won't do anything? Also
> shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim
> calls? At the moment seems any value is ignored...

This is a question for glibc developers. For that matter, this entire
thread is really about tuning of the malloc in glibc and should
ideally be addressed upstream.

There has been some discussion of this in other contexts, e.g. Python at

https://nuald.blogspot.com/2013/06/memory-reclaiming-in-python.html

and emacs at

http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html

As the Python posts poitns out, it is possible to use alternate malloc
implementations, either rebuilding R to use them or using LD_PRELOAD.
On Ubuntu for example, you can have R use jemalloc with

sudo apt-get install libjemalloc1
env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R

This does not seem to hold onto memory to the same degree, but I don't
know about any other aspect of its performance.

The emacs post suggests that calling malloc_trim may have more of an
effect in some cases: the post describes callign it via gdb on all
running processes and sees a substantial memory footprint drop for
emacs, Xorg and opera. I tried the same experiment on my system with
emacs, Xorg and firefox and didn't see what that post saw -- maybe 1
few percent recovery.

At this point I'm not sure we have enough data to show that adding
malloc_trim calls at the end of each GC, say, would warrant the
nusance of having to add configure checks. It would also be necessary
to make sure that adding this doesn't significantly ompact malloc
performance.

I don't know if this issue exists on Windows as well; it might as the
basic malloc we use there is the same as used in glibc (Dog Lea's
malloc).

Best,

luke


>
> 12 авг. 2017 г. 6:09 ПП пользователь "Simon Urbanek" <
> [hidden email]> написал:
>
>>
>> > On Aug 11, 2017, at 12:57 PM, Iñaki Úcar <[hidden email]> wrote:
>> >
>> > 2017-08-11 16:00 GMT+02:00 Martin Maechler <[hidden email]>:
>> >>>>>>> Dmitriy Selivanov <[hidden email]>
>> >>>>>>>    on Fri, 11 Aug 2017 17:33:31 +0400 writes:
>> >>
>> >>> Hi mailing list and R-core. Could someone from R-core please help me to
>> >>> create account in bugzilla? I would like to submit issue related to
>> gc() to
>> >>> wishlist.
>> >>
>> >> I will create one.
>> >>
>> >> Your previous e-mails left me pretty clueless about what the
>> >> problem is that you want to solve ... but maybe others
>> >> understand better what you mean.
>> >>
>> >> Note that in the case of such a relatively sophisticated wish
>> >> without a clear sign of a problem (in my view)
>> >> chances are not high that anything will change, unless someone
>> >> provides a (small footprint) patch towards the (R-devel aka
>> >> "trunk") sources *and* reproducible R code that depicts the
>> >> problem.
>> >
>> > How to reproduce it:
>> >
>> > a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory
>> > rm(a)
>> > gc() # the R process still has the memory assigned
>> >
>>
>> Right, but that's unavoidable because of the way Linux allocates memory -
>> see FAQ 7.42
>> The memory is free, Linux just keeps it for future allocations.
>>
>> Running malloc.trim doesn't help, because the issue is fragmentation due
>> to the linear design of the pool - you likely will have another object on
>> top so in most practical cases malloc.trim() doesn't actually do anything.
>> You can always call malloc.trim() yourself is you think it helps, but it
>> doesn't in the general case. The only way to address that would be to move
>> allocated objects from top of the pool down, but that's not something R can
>> allow, because it cannot know which code still has SEXP pointers referring
>> to that object.
>>
>> Cheers,
>> Simon
>>
>>
>>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   [hidden email]
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Dirk Eddelbuettel

On 12 August 2017 at 15:10, [hidden email] wrote:
| As the Python posts poitns out, it is possible to use alternate malloc
| implementations, either rebuilding R to use them or using LD_PRELOAD.
| On Ubuntu for example, you can have R use jemalloc with
|
| sudo apt-get install libjemalloc1
| env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R
|
| This does not seem to hold onto memory to the same degree, but I don't
| know about any other aspect of its performance.

Interesting.

I don't really know anything about malloc versus jemalloc internals but I can
affirm that redis -- an in-memory database written in single-threaded C for
high performance -- in its Debian builds has been using jemalloc for years,
presumably by choice of the maintainer. (We are very happy users of [a gently
patched] redis at work; lots of writes; very good uptime.)

Having the ability to switch to jemalloc, we could design a test bench and
compare what the impact is.

Similarly, if someone cared, I could (presumably) alter the default R build
for Debian and Ubunto to also switch to jemalloc.

Anybody feel like doing some empirics?

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Dmitriy Selivanov
Very interesting information about switching glibc malloc to jemalloc.

So I see action plan as following:

   1. set up some benchmark (need to think about design)
   2. Run it on ubuntu machine with default glibc malloc
   3. Run it with malloc_trim passed with reg.finalizer()
   4. Run it with jemalloc
   5. Review results and if they will look better than with glibc malloc -
   possibly consider switch R builds to use jemalloc on Debian, Ubuntu

Can't promise about timeline, but I will definitely try to investigate.

2017-08-13 1:36 GMT+04:00 Dirk Eddelbuettel <[hidden email]>:

>
> On 12 August 2017 at 15:10, [hidden email] wrote:
> | As the Python posts poitns out, it is possible to use alternate malloc
> | implementations, either rebuilding R to use them or using LD_PRELOAD.
> | On Ubuntu for example, you can have R use jemalloc with
> |
> | sudo apt-get install libjemalloc1
> | env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R
> |
> | This does not seem to hold onto memory to the same degree, but I don't
> | know about any other aspect of its performance.
>
> Interesting.
>
> I don't really know anything about malloc versus jemalloc internals but I
> can
> affirm that redis -- an in-memory database written in single-threaded C for
> high performance -- in its Debian builds has been using jemalloc for years,
> presumably by choice of the maintainer. (We are very happy users of [a
> gently
> patched] redis at work; lots of writes; very good uptime.)
>
> Having the ability to switch to jemalloc, we could design a test bench and
> compare what the impact is.
>
> Similarly, if someone cared, I could (presumably) alter the default R build
> for Debian and Ubunto to also switch to jemalloc.
>
> Anybody feel like doing some empirics?
>
> Dirk
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]
>



--
Regards
Dmitriy Selivanov

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Dirk Eddelbuettel

On 13 August 2017 at 15:15, Dmitriy Selivanov wrote:
| Very interesting information about switching glibc malloc to jemalloc.
|
| So I see action plan as following:
|
|    1. set up some benchmark (need to think about design)
|    2. Run it on ubuntu machine with default glibc malloc
|    3. Run it with malloc_trim passed with reg.finalizer()
|    4. Run it with jemalloc
|    5. Review results and if they will look better than with glibc malloc -
|    possibly consider switch R builds to use jemalloc on Debian, Ubuntu
|
| Can't promise about timeline, but I will definitely try to investigate.

Thumbs up!

If you set up a (public) git repo I will try to help. I have access to boxen
running these OSs ranging from 2gb ram (old netbook) to 100+gb ram (at work).

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Simon Urbanek
In reply to this post by luke-tierney
FWIW if we are talking about alternative allocators, tcmalloc is another candidate that we are using for our projects where we care about allocations and performance (another upshot is that it's very flexible so you can do a lot of cool things if you care). However I didn't try it with R - I'll have a look if it can address the issue we're talking about.


> On Aug 12, 2017, at 4:10 PM, [hidden email] wrote:
>
> On Sat, 12 Aug 2017, Dmitriy Selivanov wrote:
>
>> Strange because in my all my experiments calling malloc.trim always helped
>> - memory reported by top decreased to the level it supposed to be. Do you
>> have in mind case when calling malloc.trim won't do anything? Also
>> shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim
>> calls? At the moment seems any value is ignored...
>
> This is a question for glibc developers. For that matter, this entire
> thread is really about tuning of the malloc in glibc and should
> ideally be addressed upstream.
>
> There has been some discussion of this in other contexts, e.g. Python at
>
> https://nuald.blogspot.com/2013/06/memory-reclaiming-in-python.html
>
> and emacs at
>
> http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html
>
> As the Python posts poitns out, it is possible to use alternate malloc
> implementations, either rebuilding R to use them or using LD_PRELOAD.
> On Ubuntu for example, you can have R use jemalloc with
>
> sudo apt-get install libjemalloc1
> env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R
>
> This does not seem to hold onto memory to the same degree, but I don't
> know about any other aspect of its performance.
>
> The emacs post suggests that calling malloc_trim may have more of an
> effect in some cases: the post describes callign it via gdb on all
> running processes and sees a substantial memory footprint drop for
> emacs, Xorg and opera. I tried the same experiment on my system with
> emacs, Xorg and firefox and didn't see what that post saw -- maybe 1
> few percent recovery.
>
> At this point I'm not sure we have enough data to show that adding
> malloc_trim calls at the end of each GC, say, would warrant the
> nusance of having to add configure checks. It would also be necessary
> to make sure that adding this doesn't significantly ompact malloc
> performance.
>
> I don't know if this issue exists on Windows as well; it might as the
> basic malloc we use there is the same as used in glibc (Dog Lea's
> malloc).
>
> Best,
>
> luke
>
>
>>
>> 12 авг. 2017 г. 6:09 ПП пользователь "Simon Urbanek" <
>> [hidden email]> написал:
>>
>>>
>>> > On Aug 11, 2017, at 12:57 PM, Iñaki Úcar <[hidden email]> wrote:
>>> >
>>> > 2017-08-11 16:00 GMT+02:00 Martin Maechler <[hidden email]>:
>>> >>>>>>> Dmitriy Selivanov <[hidden email]>
>>> >>>>>>>    on Fri, 11 Aug 2017 17:33:31 +0400 writes:
>>> >>
>>> >>> Hi mailing list and R-core. Could someone from R-core please help me to
>>> >>> create account in bugzilla? I would like to submit issue related to
>>> gc() to
>>> >>> wishlist.
>>> >>
>>> >> I will create one.
>>> >>
>>> >> Your previous e-mails left me pretty clueless about what the
>>> >> problem is that you want to solve ... but maybe others
>>> >> understand better what you mean.
>>> >>
>>> >> Note that in the case of such a relatively sophisticated wish
>>> >> without a clear sign of a problem (in my view)
>>> >> chances are not high that anything will change, unless someone
>>> >> provides a (small footprint) patch towards the (R-devel aka
>>> >> "trunk") sources *and* reproducible R code that depicts the
>>> >> problem.
>>> >
>>> > How to reproduce it:
>>> >
>>> > a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory
>>> > rm(a)
>>> > gc() # the R process still has the memory assigned
>>> >
>>>
>>> Right, but that's unavoidable because of the way Linux allocates memory -
>>> see FAQ 7.42
>>> The memory is free, Linux just keeps it for future allocations.
>>>
>>> Running malloc.trim doesn't help, because the issue is fragmentation due
>>> to the linear design of the pool - you likely will have another object on
>>> top so in most practical cases malloc.trim() doesn't actually do anything.
>>> You can always call malloc.trim() yourself is you think it helps, but it
>>> doesn't in the general case. The only way to address that would be to move
>>> allocated objects from top of the pool down, but that's not something R can
>>> allow, because it cannot know which code still has SEXP pointers referring
>>> to that object.
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>>
>>
>> [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>   Actuarial Science
> 241 Schaeffer Hall                  email:   [hidden email]
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help to create bugzilla account

Steve Grubb
In reply to this post by Dirk Eddelbuettel
On Saturday, August 12, 2017 5:36:36 PM EDT Dirk Eddelbuettel wrote:

> On 12 August 2017 at 15:10, [hidden email] wrote:
> | As the Python posts poitns out, it is possible to use alternate malloc
> | implementations, either rebuilding R to use them or using LD_PRELOAD.
> | On Ubuntu for example, you can have R use jemalloc with
> |
> | sudo apt-get install libjemalloc1
> | env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R
> |
> | This does not seem to hold onto memory to the same degree, but I don't
> | know about any other aspect of its performance.
>
> Interesting.
>
> I don't really know anything about malloc versus jemalloc internals but I
> can affirm that redis -- an in-memory database written in single-threaded C
> for high performance -- in its Debian builds has been using jemalloc for
> years, presumably by choice of the maintainer. (We are very happy users of
> [a gently patched] redis at work; lots of writes; very good uptime.)
>
> Having the ability to switch to jemalloc, we could design a test bench and
> compare what the impact is.
>
> Similarly, if someone cared, I could (presumably) alter the default R build
> for Debian and Ubunto to also switch to jemalloc.

Depending on how this turns out, Fedora, RHEL, Centos also have jemalloc and
tcmalloc. Meaning, if its good on those two, its good on Linux in general.
Basically, jemalloc is faster for many work loads but its harder to spot
problems. Glibc is better at spotting memory bugs but not as fast.

-Steve

> Anybody feel like doing some empirics?
>
> Dirk

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Loading...