New lazyload rdx key type: list(eagerKey=, lazyKeys=)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

New lazyload rdx key type: list(eagerKey=, lazyKeys=)

R devel mailing list
Prior to R-3.6.0 the keys in the lazyload key files, e.g.
pkg/data/Rdata.rdx or pkg/R/pkg.rdx, seemed to all be 2-long integer
vectors.  Now they can be lists.  The ones I have seen have two components,
"eagerKey" is a 2-long integer vector and "lazyKeys" is a named list of
2-long integer vectors.

> rdx <- readRDS(system.file(package="survival", "data", "Rdata.rdx"))
> str(Filter(is.list, rdx$references))
List of 2
 $ env::1:List of 2
  ..$ eagerKey: int [1:2] 273691 183
  ..$ lazyKeys:List of 1
  .. ..$ lines: int [1:2] 273874 284
 $ env::2:List of 2
  ..$ eagerKey: int [1:2] 473142 166
  ..$ lazyKeys:List of 1
  .. ..$ lines: int [1:2] 473308 310

or

>  rdx <- readRDS(system.file(package="lambda.r", "R", "lambda.r.rdx"))
> length(Filter(is.integer, rdx$references))
[1] 4
> str(Filter(Negate(is.integer), rdx$references))
List of 5
 $ env::5:List of 2
  ..$ eagerKey: int [1:2] 28278 328
  ..$ lazyKeys:List of 2
  .. ..$ lines    : int [1:2] 28606 80
  .. ..$ parseData: int [1:2] 28686 389
 $ env::6:List of 2
  ..$ eagerKey: int [1:2] 29075 327
  ..$ lazyKeys:List of 2
  .. ..$ lines    : int [1:2] 29402 71
  .. ..$ parseData: int [1:2] 29473 321
 $ env::7:List of 2
  ..$ eagerKey: int [1:2] 29794 325
  ..$ lazyKeys:List of 2
  .. ..$ lines    : int [1:2] 30119 117
  .. ..$ parseData: int [1:2] 30236 752
... many more ...

All the ones I've seen involve the environment in srcref attributes and
most packages do not keep that.  Will these be used for more sorts of
environments in the future?

What is the meaning of the lazyKeys?  Are these stored as promises until
needed or is there some special option to never or always load them?

Bill Dunlap
TIBCO Software
wdunlap tibco.com

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: New lazyload rdx key type: list(eagerKey=, lazyKeys=)

Tomas Kalibera
On 8/31/19 12:35 AM, William Dunlap via R-devel wrote:

> Prior to R-3.6.0 the keys in the lazyload key files, e.g.
> pkg/data/Rdata.rdx or pkg/R/pkg.rdx, seemed to all be 2-long integer
> vectors.  Now they can be lists.  The ones I have seen have two components,
> "eagerKey" is a 2-long integer vector and "lazyKeys" is a named list of
> 2-long integer vectors.
>
>> rdx <- readRDS(system.file(package="survival", "data", "Rdata.rdx"))
>> str(Filter(is.list, rdx$references))
> List of 2
>   $ env::1:List of 2
>    ..$ eagerKey: int [1:2] 273691 183
>    ..$ lazyKeys:List of 1
>    .. ..$ lines: int [1:2] 273874 284
>   $ env::2:List of 2
>    ..$ eagerKey: int [1:2] 473142 166
>    ..$ lazyKeys:List of 1
>    .. ..$ lines: int [1:2] 473308 310
>
> or
>
>>   rdx <- readRDS(system.file(package="lambda.r", "R", "lambda.r.rdx"))
>> length(Filter(is.integer, rdx$references))
> [1] 4
>> str(Filter(Negate(is.integer), rdx$references))
> List of 5
>   $ env::5:List of 2
>    ..$ eagerKey: int [1:2] 28278 328
>    ..$ lazyKeys:List of 2
>    .. ..$ lines    : int [1:2] 28606 80
>    .. ..$ parseData: int [1:2] 28686 389
>   $ env::6:List of 2
>    ..$ eagerKey: int [1:2] 29075 327
>    ..$ lazyKeys:List of 2
>    .. ..$ lines    : int [1:2] 29402 71
>    .. ..$ parseData: int [1:2] 29473 321
>   $ env::7:List of 2
>    ..$ eagerKey: int [1:2] 29794 325
>    ..$ lazyKeys:List of 2
>    .. ..$ lines    : int [1:2] 30119 117
>    .. ..$ parseData: int [1:2] 30236 752
> ... many more ...
>
> All the ones I've seen involve the environment in srcref attributes and
> most packages do not keep that.  Will these be used for more sorts of
> environments in the future?

The new mechanism allows to load lazily selected bindings from an
environment - previously only the bindings from the package/namespace
environment were loaded lazily. The mechanism is currently used only for
"lines" and "parseData" bindings of "srcfile" environments. Packages
that include parse data will benefit from the latter and all packages
that include source references will benefit from both. This change made
it feasible to run package checks with source references enabled, so now
one can get more useful error traces with execution time overhead
reasonable for check runs.

The format of the lazy loading database is specific to R release, it can
change at any time. Specifically this mechanism could be used for more
environments, but I am not aware of any that would benefit at the moment.

> What is the meaning of the lazyKeys?  Are these stored as promises until
> needed or is there some special option to never or always load them?

Bindings listed in lazyKeys are treated the same way as bindings in the
package/namespace environment, so they become promises after loading the
environment. Those promises load the individual objects for the bindings
when forced (details in makeLazyLoad.R and lazyLoad.R and now also
explicitly in R-ints).

Best
Tomas

>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel