Function in default parameter value closing over variables defined later in the enclosing function

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Function in default parameter value closing over variables defined later in the enclosing function

Ivan Krylov
Hi!

I needed to generalize a loss function being optimized inside another
function, so I made it a function argument with a default value. It
worked without problems, but later I noticed that the inner function,
despite being defined in the function arguments, somehow closes over a
variable belonging to the outer function, which is defined later.

Example:

outside <- function(inside = function() print(secret)) {
        secret <- 'secret'
        inside()
}
outside()

I'm used to languages that have both lambdas and variable declaration
(like perl5 -Mstrict or C++11), so I was a bit surprised.

Does this work because R looks up the variable by name late enough at
runtime for the `secret` variable to exist in the parent environment of
the `inside` function? Can I rely on it? Is this considered bad style?
Should I rewrite it (and how)?

--
Best regards,
Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

Jan Kim
Hi Ivan & All,

R's scoping system basically goes to all environments along the call
stack when trying to resolve an unbound variable, see the language
definition [1], section 4.3.4, and perhaps also 2.1.5.

Generally, unbound variables should be used with care. It's a bit
difficult to decide whether and how the code should be rewritten,
I'd say that depends on the underlying intentions / purposes. As it
is, the code could be simplified to just

    print("secret");

but that's probably missing the point.

Best regards, Jan


[1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html

On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:

> Hi!
>
> I needed to generalize a loss function being optimized inside another
> function, so I made it a function argument with a default value. It
> worked without problems, but later I noticed that the inner function,
> despite being defined in the function arguments, somehow closes over a
> variable belonging to the outer function, which is defined later.
>
> Example:
>
> outside <- function(inside = function() print(secret)) {
> secret <- 'secret'
> inside()
> }
> outside()
>
> I'm used to languages that have both lambdas and variable declaration
> (like perl5 -Mstrict or C++11), so I was a bit surprised.
>
> Does this work because R looks up the variable by name late enough at
> runtime for the `secret` variable to exist in the parent environment of
> the `inside` function? Can I rely on it? Is this considered bad style?
> Should I rewrite it (and how)?
>
> --
> Best regards,
> Ivan
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

Duncan Murdoch-2
In reply to this post by Ivan Krylov
On 23/01/2019 4:53 a.m., Ivan Krylov wrote:

> Hi!
>
> I needed to generalize a loss function being optimized inside another
> function, so I made it a function argument with a default value. It
> worked without problems, but later I noticed that the inner function,
> despite being defined in the function arguments, somehow closes over a
> variable belonging to the outer function, which is defined later.
>
> Example:
>
> outside <- function(inside = function() print(secret)) {
> secret <- 'secret'
> inside()
> }
> outside()
>
> I'm used to languages that have both lambdas and variable declaration
> (like perl5 -Mstrict or C++11), so I was a bit surprised.

Defaults of variables are evaluated in the evaluation frame of the call.
So the inside() function is created in the evaluation frame, and it's
environment will be that frame.

When it is called it will create a new evaluation frame (empty in your
example), with a parent being its environment, i.e. the evaluation frame
from when it was created, so it will be able to see your secret variable.

If it made an assignment to secret using standard "<-" assignment, it
would create a new variable in its own evaluation frame, but if it used
superassignment "<<-", it would modify the original secret variable.

>
> Does this work because R looks up the variable by name late enough at
> runtime for the `secret` variable to exist in the parent environment of
> the `inside` function? Can I rely on it? Is this considered bad style?
> Should I rewrite it (and how)?

I would consider it bad style if the inside() function had anything
other than a trivial definition as in your example.  However, in my
opinion it would be fine to write it as

  outside <- function(inside = defaultInsideFn) {
     defaultInsideFn <- function() print(secret)
     secret <- 'secret'
     inside()
  }

which is essentially equivalent, other than having a shorter header on
the outside() function.

Duncan Murdoch

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

Duncan Murdoch-2
In reply to this post by Jan Kim
On 23/01/2019 5:27 a.m., Jan T Kim wrote:
> Hi Ivan & All,
>
> R's scoping system basically goes to all environments along the call
> stack when trying to resolve an unbound variable, see the language
> definition [1], section 4.3.4, and perhaps also 2.1.5.

You are misinterpreting that section.  It's not the call stack that is
searched, it's the chain of environments that starts with the evaluation
frame of the current function.  Those are very different.  For example,


g <- function() {
   print(secret)
}

f <- function() {
   secret <- "secret"
   g()
}

would fail, because even though secret is defined in the caller of g()
and is therefore in the call stack, that's irrelevant:  it's not in g's
evaluation frame (which has no variables) or its parent (which is the
global environment if those definitions were evaluated there).

Duncan Murdoch

>
> Generally, unbound variables should be used with care. It's a bit
> difficult to decide whether and how the code should be rewritten,
> I'd say that depends on the underlying intentions / purposes. As it
> is, the code could be simplified to just
>
>      print("secret");
>
> but that's probably missing the point.
>
> Best regards, Jan
>
>
> [1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html
>
> On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:
>> Hi!
>>
>> I needed to generalize a loss function being optimized inside another
>> function, so I made it a function argument with a default value. It
>> worked without problems, but later I noticed that the inner function,
>> despite being defined in the function arguments, somehow closes over a
>> variable belonging to the outer function, which is defined later.
>>
>> Example:
>>
>> outside <- function(inside = function() print(secret)) {
>> secret <- 'secret'
>> inside()
>> }
>> outside()
>>
>> I'm used to languages that have both lambdas and variable declaration
>> (like perl5 -Mstrict or C++11), so I was a bit surprised.
>>
>> Does this work because R looks up the variable by name late enough at
>> runtime for the `secret` variable to exist in the parent environment of
>> the `inside` function? Can I rely on it? Is this considered bad style?
>> Should I rewrite it (and how)?
>>
>> --
>> Best regards,
>> Ivan
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

R help mailing list-2
Hi Duncan,

On Wed, Jan 23, 2019 at 10:02:00AM -0500, Duncan Murdoch wrote:

> On 23/01/2019 5:27 a.m., Jan T Kim wrote:
> >Hi Ivan & All,
> >
> >R's scoping system basically goes to all environments along the call
> >stack when trying to resolve an unbound variable, see the language
> >definition [1], section 4.3.4, and perhaps also 2.1.5.
>
> You are misinterpreting that section.  It's not the call stack that is
> searched, it's the chain of environments that starts with the evaluation
> frame of the current function.  Those are very different.

yes -- I meant the environment chain but mistakenly wrote "call stack",
sorry. Thanks for pointing this out.

Best regards, Jan


> For example,
>
>
> g <- function() {
>   print(secret)
> }
>
> f <- function() {
>   secret <- "secret"
>   g()
> }
>
> would fail, because even though secret is defined in the caller of g() and
> is therefore in the call stack, that's irrelevant:  it's not in g's
> evaluation frame (which has no variables) or its parent (which is the global
> environment if those definitions were evaluated there).
>
> Duncan Murdoch
>
> >
> >Generally, unbound variables should be used with care. It's a bit
> >difficult to decide whether and how the code should be rewritten,
> >I'd say that depends on the underlying intentions / purposes. As it
> >is, the code could be simplified to just
> >
> >     print("secret");
> >
> >but that's probably missing the point.
> >
> >Best regards, Jan
> >
> >
> >[1] https://cran.r-project.org/doc/manuals/r-release/R-lang.html
> >
> >On Wed, Jan 23, 2019 at 12:53:01PM +0300, Ivan Krylov wrote:
> >>Hi!
> >>
> >>I needed to generalize a loss function being optimized inside another
> >>function, so I made it a function argument with a default value. It
> >>worked without problems, but later I noticed that the inner function,
> >>despite being defined in the function arguments, somehow closes over a
> >>variable belonging to the outer function, which is defined later.
> >>
> >>Example:
> >>
> >>outside <- function(inside = function() print(secret)) {
> >> secret <- 'secret'
> >> inside()
> >>}
> >>outside()
> >>
> >>I'm used to languages that have both lambdas and variable declaration
> >>(like perl5 -Mstrict or C++11), so I was a bit surprised.
> >>
> >>Does this work because R looks up the variable by name late enough at
> >>runtime for the `secret` variable to exist in the parent environment of
> >>the `inside` function? Can I rely on it? Is this considered bad style?
> >>Should I rewrite it (and how)?
> >>
> >>--
> >>Best regards,
> >>Ivan
> >>
> >>______________________________________________
> >>[hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>https://stat.ethz.ch/mailman/listinfo/r-help
> >>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >>and provide commented, minimal, self-contained, reproducible code.
> >
> >______________________________________________
> >[hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >https://stat.ethz.ch/mailman/listinfo/r-help
> >PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> >and provide commented, minimal, self-contained, reproducible code.
> >
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

Ivan Krylov
In reply to this post by Duncan Murdoch-2
Dear Jan & Duncan,

Thanks for your replies!

On Wed, 23 Jan 2019 09:56:25 -0500
Duncan Murdoch <[hidden email]> wrote:

> Defaults of variables are evaluated in the evaluation frame of the
> call. So the inside() function is created in the evaluation frame,
> and it's environment will be that frame.
 
> When it is called it will create a new evaluation frame (empty in
> your example), with a parent being its environment, i.e. the
> evaluation frame from when it was created, so it will be able to see
> your secret variable.

Nice explanation about closures in R inheriting not only their
explicitly captured variables, but whole environments of evaluation
(not stack) frames where they have been created.

> in my opinion it would be fine to write it as
>
>   outside <- function(inside = defaultInsideFn) {
>      defaultInsideFn <- function() print(secret)
>      secret <- 'secret'
>      inside()
>   }

I like this idea; I'm going to use it.

--
Best regards,
Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

Jeff Newmiller
My objection to this design pattern is that this gives the default implementation of inside an ability that cannot be altered using functions provided by the caller. You might think this is what you want now but it has the potential to render the code unreusable in the future, which renders the whole idea of making inside an argument to outside pointless. It would be better to also make secret an argument to outside instead of a local variable or to give up on supplying the inside function as an argument.

On January 24, 2019 6:39:49 AM PST, Ivan Krylov <[hidden email]> wrote:

>Dear Jan & Duncan,
>
>Thanks for your replies!
>
>On Wed, 23 Jan 2019 09:56:25 -0500
>Duncan Murdoch <[hidden email]> wrote:
>
>> Defaults of variables are evaluated in the evaluation frame of the
>> call. So the inside() function is created in the evaluation frame,
>> and it's environment will be that frame.
>
>> When it is called it will create a new evaluation frame (empty in
>> your example), with a parent being its environment, i.e. the
>> evaluation frame from when it was created, so it will be able to see
>> your secret variable.
>
>Nice explanation about closures in R inheriting not only their
>explicitly captured variables, but whole environments of evaluation
>(not stack) frames where they have been created.
>
>> in my opinion it would be fine to write it as
>>
>>   outside <- function(inside = defaultInsideFn) {
>>      defaultInsideFn <- function() print(secret)
>>      secret <- 'secret'
>>      inside()
>>   }
>
>I like this idea; I'm going to use it.

--
Sent from my phone. Please excuse my brevity.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Function in default parameter value closing over variables defined later in the enclosing function

Ivan Krylov
On Thu, 24 Jan 2019 06:53:20 -0800
Jeff Newmiller <[hidden email]> wrote:

> It would be better to also make secret an argument to outside instead
> of a local variable or to give up on supplying the inside function as
> an argument.

This was in a small, mostly self-contained one-off script that tested
different design of experiment approaches with simulated datasets.

Actually, I should move the "secret" variable to the global level,
together with other global settings like the dataset size and noise
level. There it's accessible to both any functions that might be
interested in it and the user who might want to change it, after all.

--
Best regards,
Ivan

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.