Possible with enableJIT function

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Possible with enableJIT function

Berend Hasselman


In this email to the R-help list: https://stat.ethz.ch/pipermail/r-help/2017-June/447474.html
and in this question on Stackoverflow: https://stackoverflow.com/questions/44486643/nleqslv-memory-use-in-r

Andrew Leach has raised a question about the memory usage of my package nleqslv.
In a model with a loop within a function he has experienced continuously increasing memory usage
by package nleqslv leading to an out of memory condition.

I have been able to reproduce the continuously increasing memory usage with the following small example.

<code>
library(nleqslv,lib.loc="../nleqslv.Rcheck")
library(pryr)
dslnex <- function(x) {
    y <- numeric(2)
    y[1] <- x[1]^2 + x[2]^2 - 2
    y[2] <- exp(x[1]-1) + x[2]^3 - 2
    y
}
xstart <- c(x1=1.5,x2=2)
nsims <- 10
for(test_iter in seq_len(nsims)){
    z <- nleqslv(xstart,dslnex,jacobian=NULL)
    print(paste("nleqslv iteration",test_iter,"and memory used is",mem_used()))
}
memory.profile()
gc()
print(paste("At end memory used is", mem_used()))
</code>

The final output is

<output>
[1] "nleqslv iteration 1 and memory used is 28921288"
[1] "nleqslv iteration 2 and memory used is 29133256"
[1] "nleqslv iteration 3 and memory used is 29132992"
[1] "nleqslv iteration 4 and memory used is 29134712"
[1] "nleqslv iteration 5 and memory used is 29136432"
[1] "nleqslv iteration 6 and memory used is 29138152"
[1] "nleqslv iteration 7 and memory used is 29139872"
[1] "nleqslv iteration 8 and memory used is 29141592"
[1] "nleqslv iteration 9 and memory used is 29143312"
[1] "nleqslv iteration 10 and memory used is 29145032"
>
> memory.profile()
       NULL      symbol    pairlist     closure environment     promise
          1        8554      194726        4230        1065        6534
   language     special     builtin        char     logical     integer
      51374          45         671        9786        8030       37258
     double     complex   character         ...         any        list
       2645           1       53795           0           0       18487
 expression    bytecode externalptr     weakref         raw          S4
          1       14662        2233         592         593        1049
> gc()
         used (Mb) gc trigger (Mb) max used (Mb)
Ncells 416340 22.3     750400 40.1   581181 31.1
Vcells 726127  5.6    1308461 10.0  1191521  9.1
> print(paste("At end memory used is", mem_used()))
[1] "At end memory used is 29126472"
</output>

Indeed memory used increases in each pass of the for loop.

I have added these two lines at the top of the code after the  library(pryr) invocation.

library(compiler)
oldJIT <- enableJIT(0)

This resolves the issue in the sense that memory used remains constant after the first iteration.
Output in the for loop is now:
<output>
[1] "nleqslv iteration 1 and memory used is 24487784"
[1] "nleqslv iteration 2 and memory used is 24495816"
[1] "nleqslv iteration 3 and memory used is 24495816"
[1] "nleqslv iteration 4 and memory used is 24495816"
[1] "nleqslv iteration 5 and memory used is 24495816"
[1] "nleqslv iteration 6 and memory used is 24495816"
[1] "nleqslv iteration 7 and memory used is 24495816"
[1] "nleqslv iteration 8 and memory used is 24495816"
[1] "nleqslv iteration 9 and memory used is 24495816"
[1] "nleqslv iteration 10 and memory used is 24495816"
</output>

My questions are

- is this a bug(let) in the JIT compiler?
- if it isn't what would need to be changed in nleqslv.R in the package source? (I haven't a clue)

regards

Berend Hasselman

My sessionInfo:

R version 3.4.0 (2017-04-21)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.5

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.4.0

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Possible with enableJIT function

luke-tierney
Thanks. This should be resolved in R-devel(r72788) and R-patched
(r72789)

Best,

luke

On Mon, 12 Jun 2017, Berend Hasselman wrote:

>
>
> In this email to the R-help list: https://stat.ethz.ch/pipermail/r-help/2017-June/447474.html
> and in this question on Stackoverflow: https://stackoverflow.com/questions/44486643/nleqslv-memory-use-in-r
>
> Andrew Leach has raised a question about the memory usage of my package nleqslv.
> In a model with a loop within a function he has experienced continuously increasing memory usage
> by package nleqslv leading to an out of memory condition.
>
> I have been able to reproduce the continuously increasing memory usage with the following small example.
>
> <code>
> library(nleqslv,lib.loc="../nleqslv.Rcheck")
> library(pryr)
> dslnex <- function(x) {
>    y <- numeric(2)
>    y[1] <- x[1]^2 + x[2]^2 - 2
>    y[2] <- exp(x[1]-1) + x[2]^3 - 2
>    y
> }
> xstart <- c(x1=1.5,x2=2)
> nsims <- 10
> for(test_iter in seq_len(nsims)){
>    z <- nleqslv(xstart,dslnex,jacobian=NULL)
>    print(paste("nleqslv iteration",test_iter,"and memory used is",mem_used()))
> }
> memory.profile()
> gc()
> print(paste("At end memory used is", mem_used()))
> </code>
>
> The final output is
>
> <output>
> [1] "nleqslv iteration 1 and memory used is 28921288"
> [1] "nleqslv iteration 2 and memory used is 29133256"
> [1] "nleqslv iteration 3 and memory used is 29132992"
> [1] "nleqslv iteration 4 and memory used is 29134712"
> [1] "nleqslv iteration 5 and memory used is 29136432"
> [1] "nleqslv iteration 6 and memory used is 29138152"
> [1] "nleqslv iteration 7 and memory used is 29139872"
> [1] "nleqslv iteration 8 and memory used is 29141592"
> [1] "nleqslv iteration 9 and memory used is 29143312"
> [1] "nleqslv iteration 10 and memory used is 29145032"
>>
>> memory.profile()
>       NULL      symbol    pairlist     closure environment     promise
>          1        8554      194726        4230        1065        6534
>   language     special     builtin        char     logical     integer
>      51374          45         671        9786        8030       37258
>     double     complex   character         ...         any        list
>       2645           1       53795           0           0       18487
> expression    bytecode externalptr     weakref         raw          S4
>          1       14662        2233         592         593        1049
>> gc()
>         used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 416340 22.3     750400 40.1   581181 31.1
> Vcells 726127  5.6    1308461 10.0  1191521  9.1
>> print(paste("At end memory used is", mem_used()))
> [1] "At end memory used is 29126472"
> </output>
>
> Indeed memory used increases in each pass of the for loop.
>
> I have added these two lines at the top of the code after the  library(pryr) invocation.
>
> library(compiler)
> oldJIT <- enableJIT(0)
>
> This resolves the issue in the sense that memory used remains constant after the first iteration.
> Output in the for loop is now:
> <output>
> [1] "nleqslv iteration 1 and memory used is 24487784"
> [1] "nleqslv iteration 2 and memory used is 24495816"
> [1] "nleqslv iteration 3 and memory used is 24495816"
> [1] "nleqslv iteration 4 and memory used is 24495816"
> [1] "nleqslv iteration 5 and memory used is 24495816"
> [1] "nleqslv iteration 6 and memory used is 24495816"
> [1] "nleqslv iteration 7 and memory used is 24495816"
> [1] "nleqslv iteration 8 and memory used is 24495816"
> [1] "nleqslv iteration 9 and memory used is 24495816"
> [1] "nleqslv iteration 10 and memory used is 24495816"
> </output>
>
> My questions are
>
> - is this a bug(let) in the JIT compiler?
> - if it isn't what would need to be changed in nleqslv.R in the package source? (I haven't a clue)
>
> regards
>
> Berend Hasselman
>
> My sessionInfo:
>
> R version 3.4.0 (2017-04-21)
> Platform: x86_64-apple-darwin15.6.0 (64-bit)
> Running under: macOS Sierra 10.12.5
>
> Matrix products: default
> BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
> LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
>
> locale:
> [1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.4.0
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   [hidden email]
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Possible with enableJIT function

Berend Hasselman

> On 13 Jun 2017, at 22:05, [hidden email] wrote:
>
> Thanks. This should be resolved in R-devel(r72788) and R-patched
> (r72789)
>

Also thanks.
It is resolved in R-patched.

Berend

> Best,
>
> luke
>
> On Mon, 12 Jun 2017, Berend Hasselman wrote:
>
>>
>>
>> In this email to the R-help list: https://stat.ethz.ch/pipermail/r-help/2017-June/447474.html
>> and in this question on Stackoverflow: https://stackoverflow.com/questions/44486643/nleqslv-memory-use-in-r
>>
>> Andrew Leach has raised a question about the memory usage of my package nleqslv.
>> In a model with a loop within a function he has experienced continuously increasing memory usage
>> by package nleqslv leading to an out of memory condition.
>>
>> I have been able to reproduce the continuously increasing memory usage with the following small example.
>>
>> <code>
>> library(nleqslv,lib.loc="../nleqslv.Rcheck")
>> library(pryr)
>> dslnex <- function(x) {
>>   y <- numeric(2)
>>   y[1] <- x[1]^2 + x[2]^2 - 2
>>   y[2] <- exp(x[1]-1) + x[2]^3 - 2
>>   y
>> }
>> xstart <- c(x1=1.5,x2=2)
>> nsims <- 10
>> for(test_iter in seq_len(nsims)){
>>   z <- nleqslv(xstart,dslnex,jacobian=NULL)
>>   print(paste("nleqslv iteration",test_iter,"and memory used is",mem_used()))
>> }
>> memory.profile()
>> gc()
>> print(paste("At end memory used is", mem_used()))
>> </code>
>>
>> The final output is
>>
>> <output>
>> [1] "nleqslv iteration 1 and memory used is 28921288"
>> [1] "nleqslv iteration 2 and memory used is 29133256"
>> [1] "nleqslv iteration 3 and memory used is 29132992"
>> [1] "nleqslv iteration 4 and memory used is 29134712"
>> [1] "nleqslv iteration 5 and memory used is 29136432"
>> [1] "nleqslv iteration 6 and memory used is 29138152"
>> [1] "nleqslv iteration 7 and memory used is 29139872"
>> [1] "nleqslv iteration 8 and memory used is 29141592"
>> [1] "nleqslv iteration 9 and memory used is 29143312"
>> [1] "nleqslv iteration 10 and memory used is 29145032"
>>>
>>> memory.profile()
>>      NULL      symbol    pairlist     closure environment     promise
>>         1        8554      194726        4230        1065        6534
>>  language     special     builtin        char     logical     integer
>>     51374          45         671        9786        8030       37258
>>    double     complex   character         ...         any        list
>>      2645           1       53795           0           0       18487
>> expression    bytecode externalptr     weakref         raw          S4
>>         1       14662        2233         592         593        1049
>>> gc()
>>        used (Mb) gc trigger (Mb) max used (Mb)
>> Ncells 416340 22.3     750400 40.1   581181 31.1
>> Vcells 726127  5.6    1308461 10.0  1191521  9.1
>>> print(paste("At end memory used is", mem_used()))
>> [1] "At end memory used is 29126472"
>> </output>
>>
>> Indeed memory used increases in each pass of the for loop.
>>
>> I have added these two lines at the top of the code after the  library(pryr) invocation.
>>
>> library(compiler)
>> oldJIT <- enableJIT(0)
>>
>> This resolves the issue in the sense that memory used remains constant after the first iteration.
>> Output in the for loop is now:
>> <output>
>> [1] "nleqslv iteration 1 and memory used is 24487784"
>> [1] "nleqslv iteration 2 and memory used is 24495816"
>> [1] "nleqslv iteration 3 and memory used is 24495816"
>> [1] "nleqslv iteration 4 and memory used is 24495816"
>> [1] "nleqslv iteration 5 and memory used is 24495816"
>> [1] "nleqslv iteration 6 and memory used is 24495816"
>> [1] "nleqslv iteration 7 and memory used is 24495816"
>> [1] "nleqslv iteration 8 and memory used is 24495816"
>> [1] "nleqslv iteration 9 and memory used is 24495816"
>> [1] "nleqslv iteration 10 and memory used is 24495816"
>> </output>
>>
>> My questions are
>>
>> - is this a bug(let) in the JIT compiler?
>> - if it isn't what would need to be changed in nleqslv.R in the package source? (I haven't a clue)
>>
>> regards
>>
>> Berend Hasselman
>>
>> My sessionInfo:
>>
>> R version 3.4.0 (2017-04-21)
>> Platform: x86_64-apple-darwin15.6.0 (64-bit)
>> Running under: macOS Sierra 10.12.5
>>
>> Matrix products: default
>> BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
>> LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
>>
>> locale:
>> [1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8
>>
>> attached base packages:
>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>
>> loaded via a namespace (and not attached):
>> [1] compiler_3.4.0
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> --
> Luke Tierney
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa                  Phone:             319-335-3386
> Department of Statistics and        Fax:               319-335-3017
>   Actuarial Science
> 241 Schaeffer Hall                  email:   [hidden email]
> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel