Transferring ownership of R-managed buffer

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Transferring ownership of R-managed buffer

Tim Keitt-3
I have a use case where I would like to create an SEXP around an existing
buffer that is managed by R, thus avoiding a copy operation. If I have
something like:

void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
... additional maniupulation ...
SEXP x = somefunc(SXPTYPE, n, p);  // ????

Is there a "placement" constructor available? (I have arranged for the
corresponding UNPROTECT.) I've looked at and experimented with R_allocator
and allocVector3, but can't quite get it right. I know this is odd, but it
makes sense for my use case.

Thanks for any pointers.

THK

http://www.keittlab.org/

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Hervé Pagès-2
Hi Tim,

On 03/29/2017 08:24 AM, Tim Keitt wrote:
> I have a use case where I would like to create an SEXP around an existing
> buffer that is managed by R, thus avoiding a copy operation.

What to you mean exactly by "an existing buffer managed by R"?

> If I have
> something like:
>
> void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
> ... additional maniupulation ...
> SEXP x = somefunc(SXPTYPE, n, p);  // ????
>
> Is there a "placement" constructor available?

What's a "placement" constructor?

>(I have arranged for the
> corresponding UNPROTECT.) I've looked at and experimented with R_allocator
> and allocVector3, but can't quite get it right. I know this is odd, but it
> makes sense for my use case.

Not sure I follow what you are trying to do. Note that an SEXP is a
pointer to a C struct called SEXPREC. I think that trying to point an
SEXPREC struct to data pointed to by an existing SEXPREC struct is very
likely to lead to a disaster.

Note that if the existing buffer managed by R is an SEXP (e.g. b) and
your code has access to this SEXP then you don't need to create another
SEXP around its data. Since SEXPs are pointers doing

   SEXP x = b;

is fine and doesn't generate any copy.

And if your code only has access to a "naked" pointer to the buffer
managed by R (e.g. to RAW(b) is the buffer is actually in an SEXP)
then why would you need to wrap it inside an SEXP?

H.

--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: [hidden email]
Phone:  (206) 667-5791
Fax:    (206) 667-1319

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Hervé Pagès-2
On 03/29/2017 11:04 AM, Hervé Pagès wrote:

> Hi Tim,
>
> On 03/29/2017 08:24 AM, Tim Keitt wrote:
>> I have a use case where I would like to create an SEXP around an existing
>> buffer that is managed by R, thus avoiding a copy operation.
>
> What to you mean exactly by "an existing buffer managed by R"?
>
>> If I have
>> something like:
>>
>> void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
>> ... additional maniupulation ...
>> SEXP x = somefunc(SXPTYPE, n, p);  // ????
>>
>> Is there a "placement" constructor available?
>
> What's a "placement" constructor?
>
>> (I have arranged for the
>> corresponding UNPROTECT.) I've looked at and experimented with
>> R_allocator
>> and allocVector3, but can't quite get it right. I know this is odd,
>> but it
>> makes sense for my use case.
>
> Not sure I follow what you are trying to do. Note that an SEXP is a
> pointer to a C struct called SEXPREC. I think that trying to point an
> SEXPREC struct to data pointed to by an existing SEXPREC struct is very
> likely to lead to a disaster.
>
> Note that if the existing buffer managed by R is an SEXP (e.g. b) and
> your code has access to this SEXP then you don't need to create another
> SEXP around its data. Since SEXPs are pointers doing
>
>   SEXP x = b;
>
> is fine and doesn't generate any copy.
>
> And if your code only has access to a "naked" pointer to the buffer
> managed by R (e.g. to RAW(b) is the buffer is actually in an SEXP)
                                ^^
                                if

sorry

H.

--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: [hidden email]
Phone:  (206) 667-5791
Fax:    (206) 667-1319

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Tim Keitt-3
In reply to this post by Hervé Pagès-2
http://www.keittlab.org/

On Wed, Mar 29, 2017 at 1:04 PM, Hervé Pagès <[hidden email]> wrote:

> Hi Tim,
>
> On 03/29/2017 08:24 AM, Tim Keitt wrote:
>
>> I have a use case where I would like to create an SEXP around an existing
>> buffer that is managed by R, thus avoiding a copy operation.
>>
>
> What to you mean exactly by "an existing buffer managed by R"?
>

Not allocated from the general heap, but rather allocated using R's
allocator and therefore returnable (hypothesis!) to the R interpreter as a
vector (not external pointer) without a copy operation.


>
> If I have
>> something like:
>>
>> void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
>> ... additional maniupulation ...
>> SEXP x = somefunc(SXPTYPE, n, p);  // ????
>>
>> Is there a "placement" constructor available?
>>
>
> What's a "placement" constructor?
>

Function that constructs an object using previously allocated memory (for
example provided by a 3rd party library). In some cases, responsibility for
further management of the buffer is transferred to the new object.


>
> (I have arranged for the
>> corresponding UNPROTECT.) I've looked at and experimented with R_allocator
>> and allocVector3, but can't quite get it right. I know this is odd, but it
>> makes sense for my use case.
>>
>
> Not sure I follow what you are trying to do. Note that an SEXP is a
> pointer to a C struct called SEXPREC. I think that trying to point an
> SEXPREC struct to data pointed to by an existing SEXPREC struct is very
> likely to lead to a disaster.
>

Possibly (although that is not what I am proposing). I'm still trying to
grok where the state resides (all in the SEXPREC or between that and the
allocator).


>
> Note that if the existing buffer managed by R is an SEXP (e.g. b) and
> your code has access to this SEXP then you don't need to create another
> SEXP around its data. Since SEXPs are pointers doing
>
>   SEXP x = b;
>
> is fine and doesn't generate any copy.
>

The buffer in this case might be resized and reallocated and the original
SEXP will have gone out of scope.


>
> And if your code only has access to a "naked" pointer to the buffer
> managed by R (e.g. to RAW(b) is the buffer is actually in an SEXP)
> then why would you need to wrap it inside an SEXP?
>

I'm working with 3rd party structures that can work with a pointer and
buffer length, but cannot track the original SEXP. I just need to construct
an SEXP (SEXPREC) and point it at the buffer, set the vector length and
data type. I was checking to see if there is an API for that (and that wont
interfere with Rcpp declarations as it pulls in parts of Rinternals.h, but
as far as I can tell, not the parts that I need to work with SEXPRECs at a
low-level).

THK


> H.
>
>
>> Thanks for any pointers.
>>
>> THK
>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keit
>> tlab.org_&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWd
>> GbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5dLU_mdGwmpIk_iUqE0dPNjqw
>> y8wiRy6hS_lWF9k&s=h04DJujKDfqzbLz4FmP3_fZ5bYS3t7UEjSwpLrW5mL0&e=
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.et
>> hz.ch_mailman_listinfo_r-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84V
>> tBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5d
>> LU_mdGwmpIk_iUqE0dPNjqwy8wiRy6hS_lWF9k&s=WpLxtWWwjoZ7TjW6oW-
>> vxE6s3LY5kZCG1H5h0xb0Bbs&e=
>>
>>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: [hidden email]
> Phone:  (206) 667-5791
> Fax:    (206) 667-1319
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Gabriel Becker
Tim,

What you're describing is a special case of the ALTREP framework/API that
Luke Tierney, Tomas Kalibera, and I are working on putting into R. See my
initial proposal to the DSC here:
https://www.r-project.org/dsc/2016/slides/customvectors.html and the
subsequent branch here: https://svn.r-project.org/R/branches/ALTREP/ where
Luke and I have merged the ideas from that proposal with work he had been
pursuing independently.  The framework is implemented and passes basic
testing (make check all minus the internet checks, from what I recall).

The concept of having a vector which is a "window" into another vector
without duplication ( which I suspect is at least related to your use-case,
though I could be wrong) is a special case of one of alt-representations I
 have implemented there.

Unfortunately we were not able to get it ready in time for this upcoming
release (it is a rather sizable change to the internals), but we hope to
merge it into R-devel not too long after (there isn't a specific timeline
currently AFAIK). I know that the release after next is quite a long wait
but unfortunately that is how it played out. So anyway, in the future this
will be easy to do, and what you are trying to do may even happen
automatically.

More comments inline.

On Wed, Mar 29, 2017 at 11:36 AM, Tim Keitt <[hidden email]> wrote:

> http://www.keittlab.org/
>
> On Wed, Mar 29, 2017 at 1:04 PM, Hervé Pagès <[hidden email]> wrote:
>
> > Hi Tim,
> >
> > On 03/29/2017 08:24 AM, Tim Keitt wrote:
> >
> >> I have a use case where I would like to create an SEXP around an
> existing
> >> buffer that is managed by R, thus avoiding a copy operation.
> >>
> >
> > What to you mean exactly by "an existing buffer managed by R"?
> >
>
> Not allocated from the general heap, but rather allocated using R's
> allocator and therefore returnable (hypothesis!) to the R interpreter as a
> vector (not external pointer) without a copy operation.


>
> >
> > If I have
> >> something like:
> >>
> >> void *p = (void*) RAW(PROTECT(Rf_allocVector(RAWSXP, n)));
> >> ... additional maniupulation ...
> >> SEXP x = somefunc(SXPTYPE, n, p);  // ????
> >>
> >> Is there a "placement" constructor available?
> >>
> >
> > What's a "placement" constructor?
> >
>
> Function that constructs an object using previously allocated memory (for
> example provided by a 3rd party library). In some cases, responsibility for
> further management of the buffer is transferred to the new object.
>

I'm not sure what you mean by this last part, but the rest is one of the
primary use cases of the ALTREP stuff.  I don't think you can do what
you're describing now. See my comments below.


>
> >
> > (I have arranged for the
> >> corresponding UNPROTECT.) I've looked at and experimented with
> R_allocator
> >> and allocVector3, but can't quite get it right. I know this is odd, but
> it
> >> makes sense for my use case.
> >>
> >
> > Not sure I follow what you are trying to do. Note that an SEXP is a
> > pointer to a C struct called SEXPREC. I think that trying to point an
> > SEXPREC struct to data pointed to by an existing SEXPREC struct is very
> > likely to lead to a disaster.
> >
>
> Possibly (although that is not what I am proposing). I'm still trying to
> grok where the state resides (all in the SEXPREC or between that and the
> allocator).
>
>
> >
> > Note that if the existing buffer managed by R is an SEXP (e.g. b) and
> > your code has access to this SEXP then you don't need to create another
> > SEXP around its data. Since SEXPs are pointers doing
> >
> >   SEXP x = b;
> >
> > is fine and doesn't generate any copy.
> >
>
> The buffer in this case might be resized and reallocated and the original
> SEXP will have gone out of scope.
>

I'm a little unclear what you're trying to do here. Traditional atomic
vectors are a SEXP header plus a C array contiguous in memory, where the
SEXP header knows the length of the array. Note that resizing the array is
AFAIK not supported (it's not part of the blessed C api) and the internal
macros that code uses to do this now will have to go away when the ALTREP
branch is merged. That is in fact one of the major reasons we couldn't get
it prepared in time for this release.



>
>
> >
> > And if your code only has access to a "naked" pointer to the buffer
> > managed by R (e.g. to RAW(b) is the buffer is actually in an SEXP)
> > then why would you need to wrap it inside an SEXP?
> >
>
> I'm working with 3rd party structures that can work with a pointer and
> buffer length, but cannot track the original SEXP. I just need to construct
> an SEXP (SEXPREC) and point it at the buffer, set the vector length and
> data type. I was checking to see if there is an API for that (and that wont
> interfere with Rcpp declarations as it pulls in parts of Rinternals.h, but
> as far as I can tell, not the parts that I need to work with SEXPRECs at a
> low-level).
>

Atomic vector SEXPRECs are (and must be) contiguous in memory (header plus
payload), so I don't think what you're wanting is possible without the
ALTREP framework being in place (decoupling header and payload is a major
motivating desire for us). I may misunderstand what you want though.

Best,
~G


> THK
>
>
> > H.
> >
> >
> >> Thanks for any pointers.
> >>
> >> THK
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.keit
> >> tlab.org_&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWd
> >> GbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5dLU_mdGwmpIk_iUqE0dPNjqw
> >> y8wiRy6hS_lWF9k&s=h04DJujKDfqzbLz4FmP3_fZ5bYS3t7UEjSwpLrW5mL0&e=
> >>
> >>         [[alternative HTML version deleted]]
> >>
> >> ______________________________________________
> >> [hidden email] mailing list
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.et
> >> hz.ch_mailman_listinfo_r-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84V
> >> tBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=Ceu5d
> >> LU_mdGwmpIk_iUqE0dPNjqwy8wiRy6hS_lWF9k&s=WpLxtWWwjoZ7TjW6oW-
> >> vxE6s3LY5kZCG1H5h0xb0Bbs&e=
> >>
> >>
> > --
> > Hervé Pagès
> >
> > Program in Computational Biology
> > Division of Public Health Sciences
> > Fred Hutchinson Cancer Research Center
> > 1100 Fairview Ave. N, M1-B514
> > P.O. Box 19024
> > Seattle, WA 98109-1024
> >
> > E-mail: [hidden email]
> > Phone:  (206) 667-5791
> > Fax:    (206) 667-1319
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel




--
Gabriel Becker, PhD
Associate Scientist (Bioinformatics)
Genentech Research

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Tim Keitt-3
On Wed, Mar 29, 2017 at 4:56 PM, Gabriel Becker <[hidden email]>
wrote:

> The concept of having a vector which is a "window" into another vector
> without duplication ( which I suspect is at least related to your use-case,
> though I could be wrong) is a special case of one of alt-representations I
>  have implemented there.
>

Nice. That is exactly what I need. I was trying out a little experiment,
but I think its not possible right now because the length of an R vector is
identical to the buffer length. I can do everything I need except force the
length of the returned R vector to be only a portion of the allocated
storage.

THK

http://www.keittlab.org/

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Gabriel Becker
On Thu, Mar 30, 2017 at 11:52 AM, Tim Keitt <[hidden email]> wrote:

>
> On Wed, Mar 29, 2017 at 4:56 PM, Gabriel Becker <[hidden email]>
> wrote:
>
>> The concept of having a vector which is a "window" into another vector
>> without duplication ( which I suspect is at least related to your use-case,
>> though I could be wrong) is a special case of one of alt-representations I
>>  have implemented there.
>>
>
> Nice. That is exactly what I need. I was trying out a little experiment,
> but I think its not possible right now because the length of an R vector is
> identical to the buffer length. I can do everything I need except force the
> length of the returned R vector to be only a portion of the allocated
> storage.
>

To be entirely accurate it's "technically possible but not allowed and
likely to stop working". I.e., it can be done in the current version of R,
but the way that you would do it is a) not part of the API so is not
supported, and b) AFAIK has a good chance of going away entirely (or at
least changing significantly) in R-devel (and thus the R release after this
coming one) during the process of merging the ALTREP branch sometime after
the upcoming release.  That was the last position on the subject I heard
from Luke Tierney, anyway.


Best,
~G


>
> THK
>
> http://www.keittlab.org/
>



--
Gabriel Becker, PhD
Associate Scientist (Bioinformatics)
Genentech Research

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transferring ownership of R-managed buffer

Tim Keitt-3
On Thu, Mar 30, 2017 at 3:15 PM, Gabriel Becker <[hidden email]>
wrote:

> technically possible


So the garbage collector knows the buffer length independently of the
SEXPREC length field? Then, yes, its doable, but as you say not advisable.

THK

http://www.keittlab.org/

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Loading...