Truncating vectors by reference in C-backend

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Truncating vectors by reference in C-backend

mbonsch
Dear all,

I want to create a function shrinkVector(x) that takes x and truncates
it to the first element without copy.
With SETLENGTH and SET_TRUELENGTH, I can achieve that.  You can find a
reproducible example below.
But the memory that was freed is not available for other objects
afterwards, except if x is a list (VECSXP). Any suggestions?

library(inline)
## define the shrinking function
shrinkVector <- cfunction(signature(x = "ANY"),
                           body = paste0("SETLENGTH(x, 1);",
                                                   "SET_TRUELENGTH(x,
1);",
                                                   
"return(R_NilValue);"))
## create a large vector that only fits into memory once
x <- 1 : 2e9
## shrink it
shrinkVector(x)
## shrinking seems to have worked
print(length(x) == 1)
# [1] TRUE
print(object.size(x))
# 48 bytes
## but I can't reuse the memory for a large x2:
x2 <- 1 : 2e8
# Error: cannot allocate vector of size 762.9 Mb
## if I remove x, it works
rm(x)
gc()
x2 <- 1 : 2e8

Thank you very much for your help.

Kind regards,
Markus Bonsch

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Truncating vectors by reference in C-backend

luke-tierney
DO NOT DO THIS!!! SETLENGTH and SET_TRUELENGTH are NOT part of the API
and are for internal use only.

Tho proper way to do this is to copy the data you want into a smaller
vector and allow the larger one to be garbage collected.

[VECSXPs are like all other vectors in that they are contigously
allocated by malloc, and there is no mprtable way to return part
a memory region allocated by malloc back to malloc].

There is experimental INTERNAL support for growable vectors that is
managed with INTERNALLY with TRUELENGTH and a bit setting, but this is
NOT, at leat not yet, intended for use by packages. The ALTREP
framework that should be available in the next release might allow
something like this to be done, but care is needed to make sure R's
copying semantics are adhered to.

Best,

luke

On Tue, 21 Nov 2017, [hidden email] wrote:

> Dear all,
>
> I want to create a function shrinkVector(x) that takes x and truncates it
> to the first element without copy.
> With SETLENGTH and SET_TRUELENGTH, I can achieve that.  You can find a
> reproducible example below.
> But the memory that was freed is not available for other objects
> afterwards, except if x is a list (VECSXP). Any suggestions?
>
> library(inline)
> ## define the shrinking function
> shrinkVector <- cfunction(signature(x = "ANY"),
>                          body = paste0("SETLENGTH(x, 1);",
>                                                  "SET_TRUELENGTH(x, 1);",
>                                                  "return(R_NilValue);"))
> ## create a large vector that only fits into memory once
> x <- 1 : 2e9
> ## shrink it
> shrinkVector(x)
> ## shrinking seems to have worked
> print(length(x) == 1)
> # [1] TRUE
> print(object.size(x))
> # 48 bytes
> ## but I can't reuse the memory for a large x2:
> x2 <- 1 : 2e8
> # Error: cannot allocate vector of size 762.9 Mb
> ## if I remove x, it works
> rm(x)
> gc()
> x2 <- 1 : 2e8
>
> Thank you very much for your help.
>
> Kind regards,
> Markus Bonsch
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   [hidden email]
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel