Coping with non-standard evaluation in R program analysis

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Coping with non-standard evaluation in R program analysis

Evan James Patterson
Hello R experts,


I plan to develop a tool for dynamic analysis of R programs. I would like to trace function calls at runtime, capturing argument and return values. Following a suggestion made some time ago on this list, my high-level implementation strategy is to rewrite the AST, augmenting call expressions with pre-call and post-call shims to capture the arguments and return value, respectively.


I can think of only one fundamental conceptual obstacle to this approach: R functions are not necessarily referentially transparent. The arguments received by a function are not values but promises. They can be evaluated directly ("standard evaluation"), after applying arbitrary syntactic transformations ("non-standard evaluation", aka NSE), or not at all. Therefore, if you peek at the values of function arguments before evaluating the function, you risk altering the semantics of the program, possibly fatally.


I'm looking for general advice about how to cope with NSE in this context. I also have some specific questions:


1) Is it possible to determine whether a given function (primitive, in R, or external) uses NSE on some or all of its arguments?


2) Is it possible to inspect the promise objects received by functions, say to determine whether they have been evaluated, without actually evaluating them? The R manual is not encouraging in this area:


https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects


Thank you,


Evan


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Coping with non-standard evaluation in R program analysis

Peter Meilstrup
For 2), it is not exposed in R's standard library but it is exposed in
the Rinternals API. A promise that is forced in normal evaluation will
have PRENV set to NULL.

Peter

On Tue, Jan 2, 2018 at 4:19 PM, Evan James Patterson
<[hidden email]> wrote:

> Hello R experts,
>
>
> I plan to develop a tool for dynamic analysis of R programs. I would like to trace function calls at runtime, capturing argument and return values. Following a suggestion made some time ago on this list, my high-level implementation strategy is to rewrite the AST, augmenting call expressions with pre-call and post-call shims to capture the arguments and return value, respectively.
>
>
> I can think of only one fundamental conceptual obstacle to this approach: R functions are not necessarily referentially transparent. The arguments received by a function are not values but promises. They can be evaluated directly ("standard evaluation"), after applying arbitrary syntactic transformations ("non-standard evaluation", aka NSE), or not at all. Therefore, if you peek at the values of function arguments before evaluating the function, you risk altering the semantics of the program, possibly fatally.
>
>
> I'm looking for general advice about how to cope with NSE in this context. I also have some specific questions:
>
>
> 1) Is it possible to determine whether a given function (primitive, in R, or external) uses NSE on some or all of its arguments?
>
>
> 2) Is it possible to inspect the promise objects received by functions, say to determine whether they have been evaluated, without actually evaluating them? The R manual is not encouraging in this area:
>
>
> https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects
>
>
> Thank you,
>
>
> Evan
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Coping with non-standard evaluation in R program analysis

Benjamin Tyner
In reply to this post by Evan James Patterson
Evan,

The pryr package provides some utilities which may be handy here. In
particular see the function: is_promise.

Also, next time please post in plaintext.

Regards
Ben

> Hello R experts,
>
>
> I plan to develop a tool for dynamic analysis of R programs. I would like to trace function calls at runtime, capturing argument and return values. Following a suggestion made some time ago on this list, my high-level implementation strategy is to rewrite the AST, augmenting call expressions with pre-call and post-call shims to capture the arguments and return value, respectively.
>
>
> I can think of only one fundamental conceptual obstacle to this approach: R functions are not necessarily referentially transparent. The arguments received by a function are not values but promises. They can be evaluated directly ("standard evaluation"), after applying arbitrary syntactic transformations ("non-standard evaluation", aka NSE), or not at all. Therefore, if you peek at the values of function arguments before evaluating the function, you risk altering the semantics of the program, possibly fatally.
>
>
> I'm looking for general advice about how to cope with NSE in this context. I also have some specific questions:
>
>
> 1) Is it possible to determine whether a given function (primitive, in R, or external) uses NSE on some or all of its arguments?
>
>
> 2) Is it possible to inspect the promise objects received by functions, say to determine whether they have been evaluated, without actually evaluating them? The R manual is not encouraging in this area:
>
>
> https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects
>
>
> Thank you,
>
>
> Evan
>
>
> [[alternative HTML version deleted]]
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> [hidden email] mailing list  DIGESTED
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ------------------------------
>
> End of R-devel Digest, Vol 179, Issue 1
> ***************************************

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Coping with non-standard evaluation in R program analysis

jan Vitek
In reply to this post by Peter Meilstrup
Hi Evan,

You may find some parts of what we are doing with genthat useful.  Genthat is a tool
for creating unit tests by recording argument and return values of calls. This is
done by instrumentation of the source code.

The git repo with the code is here https://github.com/PRL-PRG/genthat

We don’t really deal with NSE though. It could be something worth thinking about for us.
The contact for genthtat is Filip.


We are also writing a tool for analyzing promises and generating execution traces
within the R VM.  This is available in
    https://github.com/PRL-PRG/R-dyntrace

Aviral and Konrad are the contacts there.


Best,

jan






Jan Vitek, Professor
Computer Science,
Northeastern University

> On Jan 3, 2018, at 7:18 PM, Peter Meilstrup <[hidden email]> wrote:
>
> For 2), it is not exposed in R's standard library but it is exposed in
> the Rinternals API. A promise that is forced in normal evaluation will
> have PRENV set to NULL.
>
> Peter
>
> On Tue, Jan 2, 2018 at 4:19 PM, Evan James Patterson
> <[hidden email]> wrote:
>> Hello R experts,
>>
>>
>> I plan to develop a tool for dynamic analysis of R programs. I would like to trace function calls at runtime, capturing argument and return values. Following a suggestion made some time ago on this list, my high-level implementation strategy is to rewrite the AST, augmenting call expressions with pre-call and post-call shims to capture the arguments and return value, respectively.
>>
>>
>> I can think of only one fundamental conceptual obstacle to this approach: R functions are not necessarily referentially transparent. The arguments received by a function are not values but promises. They can be evaluated directly ("standard evaluation"), after applying arbitrary syntactic transformations ("non-standard evaluation", aka NSE), or not at all. Therefore, if you peek at the values of function arguments before evaluating the function, you risk altering the semantics of the program, possibly fatally.
>>
>>
>> I'm looking for general advice about how to cope with NSE in this context. I also have some specific questions:
>>
>>
>> 1) Is it possible to determine whether a given function (primitive, in R, or external) uses NSE on some or all of its arguments?
>>
>>
>> 2) Is it possible to inspect the promise objects received by functions, say to determine whether they have been evaluated, without actually evaluating them? The R manual is not encouraging in this area:
>>
>>
>> https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Promise-objects
>>
>>
>> Thank you,
>>
>>
>> Evan
>>
>>
>>       [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Coping with non-standard evaluation in R program analysis

Gabriel Becker
Evan,

You can look at what CodeDepends (on cran/ github.com/duncantl/CodeDepends
) does for nonstandard evaluation. It is a static code analyzer which
allows for custom handler functions for calls to specific functions. The
default handlers provided are "NSE-aware" (many of) the common contexts
where that occurs. It then consumes arguments to those functions as NSE and
annotates them accordingly in the result. This is a heuristic, and we don't
currently do the harder bit of attempting to figure out where those
non-standard symbols will be resolved currently, but it's potentially a
start.

As for manipulating the AST, do be somewhat careful of this. You may run
afoul of other things that are doing this under the hood based on
particular assumptions, notably the rlang stuff that backs the tidyverse
these days.  You might be safe though, it depends on what you want.

Another approach is what I in the histry package, which handles top-level
expressions in both interactive session and dynamic document weaving
contexts. Happy to talk more about this off list as I have done a fair bit
of work in the space it sounds like you're in.

Best,
~G

On Thu, Jan 4, 2018 at 12:20 AM, jan Vitek <[hidden email]> wrote:

> Hi Evan,
>
> You may find some parts of what we are doing with genthat useful.  Genthat
> is a tool
> for creating unit tests by recording argument and return values of calls.
> This is
> done by instrumentation of the source code.
>
> The git repo with the code is here https://github.com/PRL-PRG/genthat
>
> We don’t really deal with NSE though. It could be something worth thinking
> about for us.
> The contact for genthtat is Filip.
>
>
> We are also writing a tool for analyzing promises and generating execution
> traces
> within the R VM.  This is available in
>     https://github.com/PRL-PRG/R-dyntrace
>
> Aviral and Konrad are the contacts there.
>
>
> Best,
>
> jan
>
>
>
>
>
>
> Jan Vitek, Professor
> Computer Science,
> Northeastern University
>
> > On Jan 3, 2018, at 7:18 PM, Peter Meilstrup <[hidden email]>
> wrote:
> >
> > For 2), it is not exposed in R's standard library but it is exposed in
> > the Rinternals API. A promise that is forced in normal evaluation will
> > have PRENV set to NULL.
> >
> > Peter
> >
> > On Tue, Jan 2, 2018 at 4:19 PM, Evan James Patterson
> > <[hidden email]> wrote:
> >> Hello R experts,
> >>
> >>
> >> I plan to develop a tool for dynamic analysis of R programs. I would
> like to trace function calls at runtime, capturing argument and return
> values. Following a suggestion made some time ago on this list, my
> high-level implementation strategy is to rewrite the AST, augmenting call
> expressions with pre-call and post-call shims to capture the arguments and
> return value, respectively.
> >>
> >>
> >> I can think of only one fundamental conceptual obstacle to this
> approach: R functions are not necessarily referentially transparent. The
> arguments received by a function are not values but promises. They can be
> evaluated directly ("standard evaluation"), after applying arbitrary
> syntactic transformations ("non-standard evaluation", aka NSE), or not at
> all. Therefore, if you peek at the values of function arguments before
> evaluating the function, you risk altering the semantics of the program,
> possibly fatally.
> >>
> >>
> >> I'm looking for general advice about how to cope with NSE in this
> context. I also have some specific questions:
> >>
> >>
> >> 1) Is it possible to determine whether a given function (primitive, in
> R, or external) uses NSE on some or all of its arguments?
> >>
> >>
> >> 2) Is it possible to inspect the promise objects received by functions,
> say to determine whether they have been evaluated, without actually
> evaluating them? The R manual is not encouraging in this area:
> >>
> >>
> >> https://cran.r-project.org/doc/manuals/r-release/R-lang.
> html#Promise-objects
> >>
> >>
> >> Thank you,
> >>
> >>
> >> Evan
> >>
> >>
> >>       [[alternative HTML version deleted]]
> >>
> >> ______________________________________________
> >> [hidden email] mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> >
> > ______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>



--
Gabriel Becker, PhD
Scientist (Bioinformatics)
Genentech Research

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel