rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

Philip Rinn
Hi,

I'm trying to understand why/how absolute file paths are stored in .rdb[1] and
.rds[2] files during package installation. As a consequence building the same r
package in different directories does not produce identical .rdb and .rds files.

The background is that I work on reproducible builds[3] of R packages. I think
this is important from an engineering point of view but also from a scientists
perspective (that's actually my point). I want to be sure that my research results
are reproducible and therefore I need software that builds reproducible.

To investigate further I'd like to ask for some help. From what I understand so
far the lazy-load databases are built by code in
src/library/tools/R/makeLazyLoad.R. The code path is not very clear to me but the
main problem I have now is that it's hard to follow the code path used to install
a package. Could someone enlighten me by pointing me to some docs or by briefly
describing the path?

Any help/comments are very welcome.

Best,
Philip

PS: could you CC me, I'm not on the list. Thanks.

[1] at least in <PKG>/R/<PKG>.rdb and <PKG>/help/<PKG>.rdb
[2] at least in <PKG>/help/paths.rds
[3] https://reproducible-builds.org

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

jdnewmil-2
Someone might respond here anyway, but I think this is more of an R-devel question.

Anyway, as long as the package file after installation has appropriate file names for where it is installed, what does it matter what is in the files before installation?
--
Sent from my phone. Please excuse my brevity.

On April 11, 2017 9:40:39 PM PDT, Philip Rinn <[hidden email]> wrote:

>Hi,
>
>I'm trying to understand why/how absolute file paths are stored in
>.rdb[1] and
>.rds[2] files during package installation. As a consequence building
>the same r
>package in different directories does not produce identical .rdb and
>.rds files.
>
>The background is that I work on reproducible builds[3] of R packages.
>I think
>this is important from an engineering point of view but also from a
>scientists
>perspective (that's actually my point). I want to be sure that my
>research results
>are reproducible and therefore I need software that builds
>reproducible.
>
>To investigate further I'd like to ask for some help. From what I
>understand so
>far the lazy-load databases are built by code in
>src/library/tools/R/makeLazyLoad.R. The code path is not very clear to
>me but the
>main problem I have now is that it's hard to follow the code path used
>to install
>a package. Could someone enlighten me by pointing me to some docs or by
>briefly
>describing the path?
>
>Any help/comments are very welcome.
>
>Best,
>Philip
>
>PS: could you CC me, I'm not on the list. Thanks.
>
>[1] at least in <PKG>/R/<PKG>.rdb and <PKG>/help/<PKG>.rdb
>[2] at least in <PKG>/help/paths.rds
>[3] https://reproducible-builds.org
>
>______________________________________________
>[hidden email] mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

Philip Rinn
Hi,

On 12.04.2017 at 08:09, Jeff Newmiller wrote:

> Someone might respond here anyway, but I think this is more of an R-devel
> question.

I tired R-devel before[1] with no response :(.

> Anyway, as long as the package file after installation has appropriate file
> names for where it is installed, what does it matter what is in the files
> before installation?

That's actually the point. In the installed .rd[bs] files the absolute file paths
are still present but often[2] the referenced files don't even exist on the
computer the packaged is installed. My approach therefore is to replace those
absolute paths by relative[3] paths. But to do so I need to understand where
exactly those absolute paths are injected in the files - that's why I asked for
help in my fist mail - sorry for not being clear enough.

Best,
Philip


[1] https://stat.ethz.ch/pipermail/r-devel/2017-April/074016.html
[2] At least on Linux for packages installed via the distribution packages system.
And I think this also holds for prebuild windows/mac builds from cran.
[3] Relative to teh root directory of the package source


______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

jdnewmil-2
I think we are (I certainly am) going to need a more concrete example. As in, point us at a specific package and filename in this package that illustrates your concern. Such precision would also be expected on R-devel, so the lack of response may have been self-inflicted.

Are you aware of the system.file and file.path functions?  File names that YOU enter into your package code should be referenced using these functions. I have been assuming so far that you were referring to filenames picked up by R as it builds the package, which appear to be handled just fine however they are stored though I don't know the details.
--
Sent from my phone. Please excuse my brevity.

On April 19, 2017 2:40:05 PM PDT, Philip Rinn <[hidden email]> wrote:

>Hi,
>
>On 12.04.2017 at 08:09, Jeff Newmiller wrote:
>
>> Someone might respond here anyway, but I think this is more of an
>R-devel
>> question.
>
>I tired R-devel before[1] with no response :(.
>
>> Anyway, as long as the package file after installation has
>appropriate file
>> names for where it is installed, what does it matter what is in the
>files
>> before installation?
>
>That's actually the point. In the installed .rd[bs] files the absolute
>file paths
>are still present but often[2] the referenced files don't even exist on
>the
>computer the packaged is installed. My approach therefore is to replace
>those
>absolute paths by relative[3] paths. But to do so I need to understand
>where
>exactly those absolute paths are injected in the files - that's why I
>asked for
>help in my fist mail - sorry for not being clear enough.
>
>Best,
>Philip
>
>
>[1] https://stat.ethz.ch/pipermail/r-devel/2017-April/074016.html
>[2] At least on Linux for packages installed via the distribution
>packages system.
>And I think this also holds for prebuild windows/mac builds from cran.
>[3] Relative to teh root directory of the package source

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

Philip Rinn
Hi,

Am 20.04.2017 um 00:32 schrieb Jeff Newmiller:
> I think we are (I certainly am) going to need a more concrete example. As in,
> point us at a specific package and filename in this package that illustrates
> your concern. Such precision would also be expected on R-devel, so the lack of
> response may have been self-inflicted.

Ok, let's give you an example:

philip@debian:~$ wget https://cran.r-project.org/src/contrib/ald_1.1.tar.gz
philip@debian:~$ mkdir test
philip@debian:~$ mkdir test1
philip@debian:~$ cp ald_1.1.tar.gz test/
philip@debian:~$ cp ald_1.1.tar.gz test1/
philip@debian:~$ cd test
philip@debian:~/test$ R CMD INSTALL --build ald_1.1.tar.gz
philip@debian:~/test$ tar -zxvf ald_1.1_R_x86_64-pc-linux-gnu.tar.gz
ald/help/paths.rds
philip@debian:~/test$ cd ../test1
philip@debian:~/test1$ R CMD INSTALL --build ald_1.1.tar.gz
philip@debian:~/test1$ tar -zxvf ald_1.1_R_x86_64-pc-linux-gnu.tar.gz
ald/help/paths.rds
philip@debian:~/test1$ cd ..
philip@debian:~$ Rscript -e "readRDS('test/ald/help/paths.rds')"
[1] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/ALD.Rd"
[2] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/ald-package.Rd"
[3] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/likALD.Rd"
[4] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/mleALD.Rd"
[5] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/momentsALD.Rd"
attr(,"first")
[1] 46
philip@debian:~$ Rscript -e "readRDS('test1/ald/help/paths.rds')"
[1] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/ALD.Rd"
[2] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/ald-package.Rd"
[3] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/likALD.Rd"
[4] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/mleALD.Rd"
[5] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/momentsALD.Rd"
attr(,"first")
[1] 46

As you see, the file help/paths.rds contains absolute paths which doe not exist
(after compilation) on the PC. I therefore try to find a way to replace these
absolute paths by relative. My favorite output would be:

philip@debian:~$ Rscript -e "readRDS('test2/ald/help/paths.rds')"
[1] "./ald/man/ALD.Rd"
[2] "./ald/man/ald-package.Rd"
[3] "./ald/man/likALD.Rd"
[4] "./ald/man/mleALD.Rd"
[5] "./ald/man/momentsALD.Rd"
attr(,"first")
[1] 46

I just ask for help to understand where those paths got inject (see my first mail
for details). I'm willing to write/test the patches needed.

Hope that helps to understand my idea.

Best,
Philip

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

David Winsemius
In reply to this post by Philip Rinn
I would have thunk that the right list would be

https://stat.ethz.ch/mailman/listinfo/r-package-devel

Best
David

Sent from my iPhone

> On Apr 19, 2017, at 3:40 PM, Philip Rinn <[hidden email]> wrote:
>
> Hi,
>
>> On 12.04.2017 at 08:09, Jeff Newmiller wrote:
>>
>> Someone might respond here anyway, but I think this is more of an R-devel
>> question.
>
> I tired R-devel before[1] with no response :(.
>
>> Anyway, as long as the package file after installation has appropriate file
>> names for where it is installed, what does it matter what is in the files
>> before installation?
>
> That's actually the point. In the installed .rd[bs] files the absolute file paths
> are still present but often[2] the referenced files don't even exist on the
> computer the packaged is installed. My approach therefore is to replace those
> absolute paths by relative[3] paths. But to do so I need to understand where
> exactly those absolute paths are injected in the files - that's why I asked for
> help in my fist mail - sorry for not being clear enough.
>
> Best,
> Philip
>
>
> [1] https://stat.ethz.ch/pipermail/r-devel/2017-April/074016.html
> [2] At least on Linux for packages installed via the distribution packages system.
> And I think this also holds for prebuild windows/mac builds from cran.
> [3] Relative to teh root directory of the package source
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: rdb and rds files include abolute file paths / help understanding how lazy-load dbs are created

jdnewmil-2
In reply to this post by Philip Rinn
I don't think this is on topic either here or on R-package-devel... it should go back to R-devel of you can provide a compelling argument for making the change.

That said, I still feel you are chasing a non-problem. The original directories generally don't even exist on the original computer any more once the package has been created, but the files they refer to definitely do exist and R has no problem finding them.

Read the R source code,  src/library/tools/R/Rd.R. If you "fix" this not-a-problem and submit a patch to R-devel, then I suspect you will have to provide a more convincing argument for applying it than you have so far given.
--
Sent from my phone. Please excuse my brevity.

On April 19, 2017 3:56:38 PM PDT, Philip Rinn <[hidden email]> wrote:

>Hi,
>
>Am 20.04.2017 um 00:32 schrieb Jeff Newmiller:
>> I think we are (I certainly am) going to need a more concrete
>example. As in,
>> point us at a specific package and filename in this package that
>illustrates
>> your concern. Such precision would also be expected on R-devel, so
>the lack of
>> response may have been self-inflicted.
>
>Ok, let's give you an example:
>
>philip@debian:~$ wget
>https://cran.r-project.org/src/contrib/ald_1.1.tar.gz
>philip@debian:~$ mkdir test
>philip@debian:~$ mkdir test1
>philip@debian:~$ cp ald_1.1.tar.gz test/
>philip@debian:~$ cp ald_1.1.tar.gz test1/
>philip@debian:~$ cd test
>philip@debian:~/test$ R CMD INSTALL --build ald_1.1.tar.gz
>philip@debian:~/test$ tar -zxvf ald_1.1_R_x86_64-pc-linux-gnu.tar.gz
>ald/help/paths.rds
>philip@debian:~/test$ cd ../test1
>philip@debian:~/test1$ R CMD INSTALL --build ald_1.1.tar.gz
>philip@debian:~/test1$ tar -zxvf ald_1.1_R_x86_64-pc-linux-gnu.tar.gz
>ald/help/paths.rds
>philip@debian:~/test1$ cd ..
>philip@debian:~$ Rscript -e "readRDS('test/ald/help/paths.rds')"
>[1] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/ALD.Rd"
>[2] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/ald-package.Rd"
>[3] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/likALD.Rd"
>[4] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/mleALD.Rd"
>[5] "/tmp/RtmpXhF0y3/R.INSTALLd717b7b387a/ald/man/momentsALD.Rd"
>attr(,"first")
>[1] 46
>philip@debian:~$ Rscript -e "readRDS('test1/ald/help/paths.rds')"
>[1] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/ALD.Rd"
>[2] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/ald-package.Rd"
>[3] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/likALD.Rd"
>[4] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/mleALD.Rd"
>[5] "/tmp/RtmpqjNKva/R.INSTALLe362823e45a/ald/man/momentsALD.Rd"
>attr(,"first")
>[1] 46
>
>As you see, the file help/paths.rds contains absolute paths which doe
>not exist
>(after compilation) on the PC. I therefore try to find a way to replace
>these
>absolute paths by relative. My favorite output would be:
>
>philip@debian:~$ Rscript -e "readRDS('test2/ald/help/paths.rds')"
>[1] "./ald/man/ALD.Rd"
>[2] "./ald/man/ald-package.Rd"
>[3] "./ald/man/likALD.Rd"
>[4] "./ald/man/mleALD.Rd"
>[5] "./ald/man/momentsALD.Rd"
>attr(,"first")
>[1] 46
>
>I just ask for help to understand where those paths got inject (see my
>first mail
>for details). I'm willing to write/test the patches needed.
>
>Hope that helps to understand my idea.
>
>Best,
>Philip

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.