Registration of native routines

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Registration of native routines

Prof Brian Ripley
Registration of 'native routines' (entry points in compiled code loaded
into R) has been available for over 14 years, but few packages make use
of it (less than 10% of those on CRAN with compiled code).

Registration has similar benefits to name spaces in R code:

- it ensures that the routines used by .C, .Call etc are those in your
package (without needing a PACKAGE argument).

- it avoids polluting the search space for native routines with those
from your package.

- it checks the number of arguments passed to .Call/.External, and the
number and optionally the type for .C/.Fortran.

- it finds native routines faster, especially if 10s of name spaces are
loaded.

Kurt Hornik and I have written a tool to make adding registration much
easier.  From NEWS in R-devel

     • Package tools has a new function
       package_native_routine_registration_skeleton() to assist adding
       native-routine registration to a package.  See its help and §5.4.1
       of ‘Writing R Extensions’ for how to use it.  (At the time it was
       added it successfully automated adding registration to over 90%
       of CRAN packages which lacked it.  Many of the failures were
       newly-detected bugs in the packages, e.g. 50 packages called
       entry points with varying numbers of arguments and 65 packages
       called entry points not in the package.)

Of the 2450 CRAN packages with compiled code but not registration, this
tool successfully[*] converts all but 220 out-of-the-box.  Another 25
packages already use R_init_pkgname and so need the registration info to
be merged.

Most of the rest fail because of errors in the package but some try to
do tricky things computing names of routines, and this is noted in the
skeleton output.

A few of you are using the older mechanism of specifying entry points in
a useDynLib call in the NAMESPACE file. This shares the first and fourth
benefits (but registration is faster), but not the second and third.

$5.4 of the 'Writing R Extensions' manual has been re-written with a
worked example of adding registration.

Shortly, R CMD check --as-cran will note if registration is not fully
used.  Expect to be asked to add registration: as increasingly large
numbers of packages are used, avoiding polluting the search space is
becoming important.

(There are reports that the check in 'R CMD check' on Windows sometimes
fails to detect use of registration.  This is being looked into:
meanwhile say so in a CRAN submission if it happens to you.)


[*] R CMD check output is unchanged.

--
Brian D. Ripley,                  [hidden email]
Emeritus Professor of Applied Statistics, University of Oxford

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Registration of native routines

Avraham Adler
On Tue, Feb 14, 2017 at 11:25 AM, Prof Brian Ripley
<[hidden email]> wrote:

> Registration of 'native routines' (entry points in compiled code loaded into
> R) has been available for over 14 years, but few packages make use of it
> (less than 10% of those on CRAN with compiled code).
>
> Registration has similar benefits to name spaces in R code:
>
> - it ensures that the routines used by .C, .Call etc are those in your
> package (without needing a PACKAGE argument).
>
> - it avoids polluting the search space for native routines with those from
> your package.
>
> - it checks the number of arguments passed to .Call/.External, and the
> number and optionally the type for .C/.Fortran.
>
> - it finds native routines faster, especially if 10s of name spaces are
> loaded.
>
> Kurt Hornik and I have written a tool to make adding registration much
> easier.  From NEWS in R-devel
>
>     • Package tools has a new function
>       package_native_routine_registration_skeleton() to assist adding
>       native-routine registration to a package.  See its help and §5.4.1
>       of ‘Writing R Extensions’ for how to use it.  (At the time it was
>       added it successfully automated adding registration to over 90%
>       of CRAN packages which lacked it.  Many of the failures were
>       newly-detected bugs in the packages, e.g. 50 packages called
>       entry points with varying numbers of arguments and 65 packages
>       called entry points not in the package.)

Hello, Dr., Ripley.

This is fantastic. Is there a way to install this functionality into
an existing 3.3.2 installation, or is it exclusive to
R-deve;/R-3.4-to-be?

Thank you,

Avi

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Registration of native routines

Prof Brian Ripley
On 14/02/2017 17:28, Avraham Adler wrote:

> On Tue, Feb 14, 2017 at 11:25 AM, Prof Brian Ripley
> <[hidden email]> wrote:
>> Registration of 'native routines' (entry points in compiled code loaded into
>> R) has been available for over 14 years, but few packages make use of it
>> (less than 10% of those on CRAN with compiled code).
>>
>> Registration has similar benefits to name spaces in R code:
>>
>> - it ensures that the routines used by .C, .Call etc are those in your
>> package (without needing a PACKAGE argument).
>>
>> - it avoids polluting the search space for native routines with those from
>> your package.
>>
>> - it checks the number of arguments passed to .Call/.External, and the
>> number and optionally the type for .C/.Fortran.
>>
>> - it finds native routines faster, especially if 10s of name spaces are
>> loaded.
>>
>> Kurt Hornik and I have written a tool to make adding registration much
>> easier.  From NEWS in R-devel
>>
>>     • Package tools has a new function
>>       package_native_routine_registration_skeleton() to assist adding
>>       native-routine registration to a package.  See its help and §5.4.1
>>       of ‘Writing R Extensions’ for how to use it.  (At the time it was
>>       added it successfully automated adding registration to over 90%
>>       of CRAN packages which lacked it.  Many of the failures were
>>       newly-detected bugs in the packages, e.g. 50 packages called
>>       entry points with varying numbers of arguments and 65 packages
>>       called entry points not in the package.)
>
> Hello, Dr., Ripley.
>
> This is fantastic. Is there a way to install this functionality into
> an existing 3.3.2 installation, or is it exclusive to
> R-deve;/R-3.4-to-be?

You need to run the tool in R-devel to get the skeleton for your
package.  Everything else will work in any recent version of R.

>
> Thank you,
>
> Avi
>


--
Brian D. Ripley,                  [hidden email]
Emeritus Professor of Applied Statistics, University of Oxford

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Registration of native routines

Prof Brian Ripley
In reply to this post by Prof Brian Ripley
On 14/02/2017 16:25, Prof Brian Ripley wrote:
> Registration of 'native routines' (entry points in compiled code loaded
> into R) has been available for over 14 years,

...

> (There are reports that the check in 'R CMD check' on Windows sometimes
> fails to detect use of registration.  This is being looked into:
> meanwhile say so in a CRAN submission if it happens to you.)

One instance has been resolved: for at least 5 years R CMD check for a
package with compiled code has assumed commands 'nm' and (on Windows)
'objdump' were on the PATH: these are also needed to detect use of
registration.  There is now a warning if they are not found.

--
Brian D. Ripley,                  [hidden email]
Emeritus Professor of Applied Statistics, University of Oxford

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Registration of native routines

Henric Winell
On 2017-02-19 09:19, Prof Brian Ripley wrote:

> On 14/02/2017 16:25, Prof Brian Ripley wrote:
>> Registration of 'native routines' (entry points in compiled code loaded
>> into R) has been available for over 14 years,
>
> ...
>
>> (There are reports that the check in 'R CMD check' on Windows sometimes
>> fails to detect use of registration.  This is being looked into:
>> meanwhile say so in a CRAN submission if it happens to you.)
>
> One instance has been resolved: for at least 5 years R CMD check for a
> package with compiled code has assumed commands 'nm' and (on Windows)
> 'objdump' were on the PATH: these are also needed to detect use of
> registration.  There is now a warning if they are not found.

Many thanks for this!  However, I think that there's a subtlety here:

On a newly installed Windows 10 64-bit system, with the latest R-devel
and Rtools, 'R CMD check' failed to detect use of registration despite
the fact that both 'nm' and 'objdump' were on the path.  It turned out
that I had the 32-bit versions on the path, i.e.,
'\\...\Rtools\mingw_32\bin', which, I believe, is the default setting in
the Rtools installer.

After switching to the 64-bit versions, the use of registration now
seems to be reliably detected.


Henric Winell

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Registration of native routines

Jeroen Ooms.
In reply to this post by Prof Brian Ripley
On Tue, Feb 14, 2017 at 5:25 PM, Prof Brian Ripley
<[hidden email]> wrote:
>
> Registration has similar benefits to name spaces in R code:
>
> - it ensures that the routines used by .C, .Call etc are those in your package (without needing a PACKAGE argument).
> - it avoids polluting the search space for native routines with those from your package.
> - it checks the number of arguments passed to .Call/.External, and the number and optionally the type for .C/.Fortran.
> - it finds native routines faster, especially if 10s of name spaces are loaded.

Do these benefits also hold for packages that currently use useDynLib
exclusively in combination symbol names? E.g for the example from WRE:

   useDynLib(foo, myRoutine, myOtherRoutine)

Which is invoked via:

  .Call(myRoutine, x, y)

What ambiguity or pollution is introduced by foo:::myRoutine? Should
manually registering 'myRoutine' in C still be mandatory in this case?

It was nice having the NAMESPACE as the central declaration of
callable C routines. The "R_registerRoutines" system will require
maintaining additional C code which re-declares all callable C
functions from other compilation units. This introduces additional
complexity for package authors and might become a source of bugs when
we forget to update the registrations when C functions have changed.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Registration of native routines

Avraham Adler
I believe Dr. Ripley said that the UseDynLib doesn't address the
middle two points of the list.

I experimented with the Delaporte package timing calls using both
methods, and they were within 1~2 microseconds of each other using
microbenchmark. Since the more complicated way is now expressly
preferred, it isn't a big deal for me personally to change so the next
version of Delaporte will use CallMethod/useDynLib(foo,
.registration=TRUE) etc. I can see how in more complicated packages
that could become more burdensome.

Thanks,

Avi

On Wed, Feb 22, 2017 at 9:34 AM, Jeroen Ooms <[hidden email]> wrote:

> On Tue, Feb 14, 2017 at 5:25 PM, Prof Brian Ripley
> <[hidden email]> wrote:
>>
>> Registration has similar benefits to name spaces in R code:
>>
>> - it ensures that the routines used by .C, .Call etc are those in your package (without needing a PACKAGE argument).
>> - it avoids polluting the search space for native routines with those from your package.
>> - it checks the number of arguments passed to .Call/.External, and the number and optionally the type for .C/.Fortran.
>> - it finds native routines faster, especially if 10s of name spaces are loaded.
>
> Do these benefits also hold for packages that currently use useDynLib
> exclusively in combination symbol names? E.g for the example from WRE:
>
>    useDynLib(foo, myRoutine, myOtherRoutine)
>
> Which is invoked via:
>
>   .Call(myRoutine, x, y)
>
> What ambiguity or pollution is introduced by foo:::myRoutine? Should
> manually registering 'myRoutine' in C still be mandatory in this case?
>
> It was nice having the NAMESPACE as the central declaration of
> callable C routines. The "R_registerRoutines" system will require
> maintaining additional C code which re-declares all callable C
> functions from other compilation units. This introduces additional
> complexity for package authors and might become a source of bugs when
> we forget to update the registrations when C functions have changed.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel