S3 lookup rules changed in R 3.6.1

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

S3 lookup rules changed in R 3.6.1

Konrad Rudolph-2
tl;dr: S3 lookup no longer works in custom non-namespace environments as of
R 3.6.1. Is this a bug?

I am implementing S3 dispatch for generic methods in environments that are
not
packages. I am trying to emulate the R package namespace mechanism by
having a
“namespace” environment that defines generics and methods, but only exposes
the
generics themselves, not the methods.

To make S3 lookup work when using the generics, I am using
`registerS3method`.
While this method itself has no extensive documentation, the documentation
of
`UseMethod` contains this relevant passage:

> Namespaces can register methods for generic functions. To support this,
> ‘UseMethod’ and ‘NextMethod’ search for methods in two places: in the
> environment in which the generic function is called, and in the
registration
> data base for the environment in which the generic is defined (typically a
> namespace). So methods for a generic function need to be available in the
> environment of the call to the generic, or they must be registered. (It
does
> not matter whether they are visible in the environment in which the
generic is
> defined.) As from R 3.5.0, the registration data base is searched after
the
> top level environment (see ‘topenv’) of the calling environment (but
before
> the parents of the top level environment).

This used to work but it stopped working in R 3.6.1 and I cannot figure out
(a)
why, and (b) how to fix it. Unfortunately I am unable to find the relevant
information by reading the R source code, even when “diff”ing what seem to
be
the only even remotely relevant changes [1].

The R NEWS merely list the following change for R 3.6.0:

> * S3method() directives in ‘NAMESPACE’ can now also be used to perform
delayed
>   S3 method registration.
> […]
> * Method dispatch uses more relevant environments when looking up class
>   definitions.

Unfortunately it is not clear to me what exactly this means.

Here’s a minimal example code that works under R 3.5.3 but breaks under
R 3.6.1
(I don’t know about 3.6.0).

```
# Define “package namespace”:
ns = new.env(parent = .BaseNamespaceEnv)
local(envir = ns, {
    test = function (x) UseMethod('test')
    test.default = function (x) message('test.default')
    test.foo = function (x) message('test.foo')

    .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
    .__S3MethodsTable__.$test.default = test.default
    .__S3MethodsTable__.$test.foo = test.foo

    # Or, equivalently:
    # registerS3method('test', 'default', test.default)
    # registerS3method('test', 'foo', test.foo)
})

# Expose generic publicly:
test = ns$test

# Usage:
test(1)
test(structure(1, class = 'foo'))
```

Output in R up to 3.5.3:

```
test.default
test.foo
```

Output in R 3.6.1:

```
Error in UseMethod("test") :
  no applicable method for 'test' applied to an object of class
"c('double', 'numeric')"
```

It’s worth noting that the output of `.S3methods` is the same for all R
versions, and from my understanding of its output, this *should* indicate
that
S3 lookup should behave identically, too. Furthermore, lookup via
`getS3method`
succeeds in all R versions, and (again, in my understanding) the logic of
this
function should be identical to the logic of R’s internal S3 dispatch:

```
getS3method('test', 'default')(1)
getS3method('test', 'foo')(1)
```

Conversely, specialising an existing generic from a loaded package works.
E.g.:

```
local(envir = ns, {
    print.foo = function (x) message('print.foo')
    registerS3method('print', 'foo', print.foo)
})

print(structure(1, class = 'foo'))
```

This prints “print.foo” in all R versions as expected.

So my question is: Why do the `test(…)` calls in R 3.6.1 no longer trigger
S3
method lookup in the generic function’s environment? Is this behaviour by
design
or is it a bug? If it’s by design, why does `getS3method` still use the old
behaviour? And, most importantly, how can I fix my definition of `ns` to
make
S3 dispatch for non-exposed methods work again?

… actually I just found a workaround:

```
ns$.packageName = 'not important'
```

This marks `ns` as a package namespace. To me, the documentation seems to
imply
that this shouldn’t be necessary (and it previously wasn’t). Furthermore,
the
code for `registerS3method` explicitly supports non-package namespace
environments. Unfortunately this workaround is not satisfactory because
pretending that the environment is a package namespace, when it really
isn’t,
might break other things.

[1] See r75273; there’s also r74625, which changes the actual lookup
mechanism
    used by `UseMethod`, but that seems even less relevant, because it is
    disabled unless a specific environment variable is set.

--
Konrad Rudolph

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: S3 lookup rules changed in R 3.6.1

Duncan Murdoch-2
On 09/10/2019 3:22 p.m., Konrad Rudolph wrote:
> tl;dr: S3 lookup no longer works in custom non-namespace environments as of
> R 3.6.1. Is this a bug?

I don't know whether this was intentional or not, but a binary search
through the svn commits finds that the errors started in this one:

------------------------------------------------------------------------
r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines
Changed paths:
    M /trunk/src/main/objects.c
    M /trunk/tests/reg-tests-1a.R

Have S3 methods lookup by default look for the S3 registry in the topenv
of the generic.
------------------------------------------------------------------------

Duncan Murdoch

>
> I am implementing S3 dispatch for generic methods in environments that are
> not
> packages. I am trying to emulate the R package namespace mechanism by
> having a
> “namespace” environment that defines generics and methods, but only exposes
> the
> generics themselves, not the methods.
>
> To make S3 lookup work when using the generics, I am using
> `registerS3method`.
> While this method itself has no extensive documentation, the documentation
> of
> `UseMethod` contains this relevant passage:
>
>> Namespaces can register methods for generic functions. To support this,
>> ‘UseMethod’ and ‘NextMethod’ search for methods in two places: in the
>> environment in which the generic function is called, and in the
> registration
>> data base for the environment in which the generic is defined (typically a
>> namespace). So methods for a generic function need to be available in the
>> environment of the call to the generic, or they must be registered. (It
> does
>> not matter whether they are visible in the environment in which the
> generic is
>> defined.) As from R 3.5.0, the registration data base is searched after
> the
>> top level environment (see ‘topenv’) of the calling environment (but
> before
>> the parents of the top level environment).
>
> This used to work but it stopped working in R 3.6.1 and I cannot figure out
> (a)
> why, and (b) how to fix it. Unfortunately I am unable to find the relevant
> information by reading the R source code, even when “diff”ing what seem to
> be
> the only even remotely relevant changes [1].
>
> The R NEWS merely list the following change for R 3.6.0:
>
>> * S3method() directives in ‘NAMESPACE’ can now also be used to perform
> delayed
>>    S3 method registration.
>> […]
>> * Method dispatch uses more relevant environments when looking up class
>>    definitions.
>
> Unfortunately it is not clear to me what exactly this means.
>
> Here’s a minimal example code that works under R 3.5.3 but breaks under
> R 3.6.1
> (I don’t know about 3.6.0).
>
> ```
> # Define “package namespace”:
> ns = new.env(parent = .BaseNamespaceEnv)
> local(envir = ns, {
>      test = function (x) UseMethod('test')
>      test.default = function (x) message('test.default')
>      test.foo = function (x) message('test.foo')
>
>      .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
>      .__S3MethodsTable__.$test.default = test.default
>      .__S3MethodsTable__.$test.foo = test.foo
>
>      # Or, equivalently:
>      # registerS3method('test', 'default', test.default)
>      # registerS3method('test', 'foo', test.foo)
> })
>
> # Expose generic publicly:
> test = ns$test
>
> # Usage:
> test(1)
> test(structure(1, class = 'foo'))
> ```
>
> Output in R up to 3.5.3:
>
> ```
> test.default
> test.foo
> ```
>
> Output in R 3.6.1:
>
> ```
> Error in UseMethod("test") :
>    no applicable method for 'test' applied to an object of class
> "c('double', 'numeric')"
> ```
>
> It’s worth noting that the output of `.S3methods` is the same for all R
> versions, and from my understanding of its output, this *should* indicate
> that
> S3 lookup should behave identically, too. Furthermore, lookup via
> `getS3method`
> succeeds in all R versions, and (again, in my understanding) the logic of
> this
> function should be identical to the logic of R’s internal S3 dispatch:
>
> ```
> getS3method('test', 'default')(1)
> getS3method('test', 'foo')(1)
> ```
>
> Conversely, specialising an existing generic from a loaded package works.
> E.g.:
>
> ```
> local(envir = ns, {
>      print.foo = function (x) message('print.foo')
>      registerS3method('print', 'foo', print.foo)
> })
>
> print(structure(1, class = 'foo'))
> ```
>
> This prints “print.foo” in all R versions as expected.
>
> So my question is: Why do the `test(…)` calls in R 3.6.1 no longer trigger
> S3
> method lookup in the generic function’s environment? Is this behaviour by
> design
> or is it a bug? If it’s by design, why does `getS3method` still use the old
> behaviour? And, most importantly, how can I fix my definition of `ns` to
> make
> S3 dispatch for non-exposed methods work again?
>
> … actually I just found a workaround:
>
> ```
> ns$.packageName = 'not important'
> ```
>
> This marks `ns` as a package namespace. To me, the documentation seems to
> imply
> that this shouldn’t be necessary (and it previously wasn’t). Furthermore,
> the
> code for `registerS3method` explicitly supports non-package namespace
> environments. Unfortunately this workaround is not satisfactory because
> pretending that the environment is a package namespace, when it really
> isn’t,
> might break other things.
>
> [1] See r75273; there’s also r74625, which changes the actual lookup
> mechanism
>      used by `UseMethod`, but that seems even less relevant, because it is
>      disabled unless a specific environment variable is set.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: S3 lookup rules changed in R 3.6.1

Konrad Rudolph-2
Oh, I had missed that that code path is now enabled by default. It’s worth
noting that the commented-out test in that commit also still succeeds if
invoked via `getS3method`. So at the very least there’s now an
inconsistency in the lookup performed by R internally (via `UseMethod`) and
`getS3method`, which is probably unintentional.

I see how the change is beneficial by preventing surprising behaviour in a
corner case. Unfortunately it also breaks at least one published package
[1], and if I understand correctly it no longer conforms to the documented
behaviour (quoted in my initial message), which even explicitly mentions
non-namespace environments.

[1] https://github.com/klmr/modules/issues/147

On Wed, Oct 9, 2019 at 11:23 PM Duncan Murdoch <[hidden email]>
wrote:

> On 09/10/2019 3:22 p.m., Konrad Rudolph wrote:
> > tl;dr: S3 lookup no longer works in custom non-namespace environments as
> of
> > R 3.6.1. Is this a bug?
>
> I don't know whether this was intentional or not, but a binary search
> through the svn commits finds that the errors started in this one:
>
> ------------------------------------------------------------------------
> r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines
> Changed paths:
>     M /trunk/src/main/objects.c
>     M /trunk/tests/reg-tests-1a.R
>
> Have S3 methods lookup by default look for the S3 registry in the topenv
> of the generic.
> ------------------------------------------------------------------------
>
> Duncan Murdoch
>
> >
> > I am implementing S3 dispatch for generic methods in environments that
> are
> > not
> > packages. I am trying to emulate the R package namespace mechanism by
> > having a
> > “namespace” environment that defines generics and methods, but only
> exposes
> > the
> > generics themselves, not the methods.
> >
> > To make S3 lookup work when using the generics, I am using
> > `registerS3method`.
> > While this method itself has no extensive documentation, the
> documentation
> > of
> > `UseMethod` contains this relevant passage:
> >
> >> Namespaces can register methods for generic functions. To support this,
> >> ‘UseMethod’ and ‘NextMethod’ search for methods in two places: in the
> >> environment in which the generic function is called, and in the
> > registration
> >> data base for the environment in which the generic is defined
> (typically a
> >> namespace). So methods for a generic function need to be available in
> the
> >> environment of the call to the generic, or they must be registered. (It
> > does
> >> not matter whether they are visible in the environment in which the
> > generic is
> >> defined.) As from R 3.5.0, the registration data base is searched after
> > the
> >> top level environment (see ‘topenv’) of the calling environment (but
> > before
> >> the parents of the top level environment).
> >
> > This used to work but it stopped working in R 3.6.1 and I cannot figure
> out
> > (a)
> > why, and (b) how to fix it. Unfortunately I am unable to find the
> relevant
> > information by reading the R source code, even when “diff”ing what seem
> to
> > be
> > the only even remotely relevant changes [1].
> >
> > The R NEWS merely list the following change for R 3.6.0:
> >
> >> * S3method() directives in ‘NAMESPACE’ can now also be used to perform
> > delayed
> >>    S3 method registration.
> >> […]
> >> * Method dispatch uses more relevant environments when looking up class
> >>    definitions.
> >
> > Unfortunately it is not clear to me what exactly this means.
> >
> > Here’s a minimal example code that works under R 3.5.3 but breaks under
> > R 3.6.1
> > (I don’t know about 3.6.0).
> >
> > ```
> > # Define “package namespace”:
> > ns = new.env(parent = .BaseNamespaceEnv)
> > local(envir = ns, {
> >      test = function (x) UseMethod('test')
> >      test.default = function (x) message('test.default')
> >      test.foo = function (x) message('test.foo')
> >
> >      .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
> >      .__S3MethodsTable__.$test.default = test.default
> >      .__S3MethodsTable__.$test.foo = test.foo
> >
> >      # Or, equivalently:
> >      # registerS3method('test', 'default', test.default)
> >      # registerS3method('test', 'foo', test.foo)
> > })
> >
> > # Expose generic publicly:
> > test = ns$test
> >
> > # Usage:
> > test(1)
> > test(structure(1, class = 'foo'))
> > ```
> >
> > Output in R up to 3.5.3:
> >
> > ```
> > test.default
> > test.foo
> > ```
> >
> > Output in R 3.6.1:
> >
> > ```
> > Error in UseMethod("test") :
> >    no applicable method for 'test' applied to an object of class
> > "c('double', 'numeric')"
> > ```
> >
> > It’s worth noting that the output of `.S3methods` is the same for all R
> > versions, and from my understanding of its output, this *should* indicate
> > that
> > S3 lookup should behave identically, too. Furthermore, lookup via
> > `getS3method`
> > succeeds in all R versions, and (again, in my understanding) the logic of
> > this
> > function should be identical to the logic of R’s internal S3 dispatch:
> >
> > ```
> > getS3method('test', 'default')(1)
> > getS3method('test', 'foo')(1)
> > ```
> >
> > Conversely, specialising an existing generic from a loaded package works.
> > E.g.:
> >
> > ```
> > local(envir = ns, {
> >      print.foo = function (x) message('print.foo')
> >      registerS3method('print', 'foo', print.foo)
> > })
> >
> > print(structure(1, class = 'foo'))
> > ```
> >
> > This prints “print.foo” in all R versions as expected.
> >
> > So my question is: Why do the `test(…)` calls in R 3.6.1 no longer
> trigger
> > S3
> > method lookup in the generic function’s environment? Is this behaviour by
> > design
> > or is it a bug? If it’s by design, why does `getS3method` still use the
> old
> > behaviour? And, most importantly, how can I fix my definition of `ns` to
> > make
> > S3 dispatch for non-exposed methods work again?
> >
> > … actually I just found a workaround:
> >
> > ```
> > ns$.packageName = 'not important'
> > ```
> >
> > This marks `ns` as a package namespace. To me, the documentation seems to
> > imply
> > that this shouldn’t be necessary (and it previously wasn’t). Furthermore,
> > the
> > code for `registerS3method` explicitly supports non-package namespace
> > environments. Unfortunately this workaround is not satisfactory because
> > pretending that the environment is a package namespace, when it really
> > isn’t,
> > might break other things.
> >
> > [1] See r75273; there’s also r74625, which changes the actual lookup
> > mechanism
> >      used by `UseMethod`, but that seems even less relevant, because it
> is
> >      disabled unless a specific environment variable is set.
> >
>
>

--
Konrad Rudolph

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel