best way to build from Git

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

best way to build from Git

frederik-2
Dear R-devel,

I checked out a recent copy of R via Subversion and made a few changes to the code. I wanted to commit them locally to my repo, just to stay organized and keep them separate from other changes I plan to make. However, I was not able to commit them because I don't think SVN allows this?

Plan B, check out a Git mirror of R, https://github.com/wch/r-source/

It has some instructions for building: https://github.com/wch/r-source/wiki

However, there are some complicated workarounds due to a commit from 2013 (https://github.com/wch/r-source/commit/4f13e5325dfbcb9fc8f55fc6027af9ae9c7750a3). And it only seems to succeed if the build directory is the same as the source directory.

Makefile.in has some odd stuff that might be related, like why is it checking for "$(srcdir)/doc/FAQ" rather than "$(top_builddir)/doc/FAQ"? Conversely, there is a check for "$(top_builddir)/.git" which I would expect to be in "$(srcdir)/.git" instead. And it runs "git svn info" which seems to take indefinitely long - I am not using "git svn", I'm just trying to build from a regular git repo. This is difficult to debug because it is inside an @if, so "make -n" still runs it. Also there are no comments to explain what e.g. the "svnonly" target is doing, or anything else.

After some time I still have little idea why it is necessary for the build system to depend on the revision control system. Clearly we are trying to do something fancy, but why not e.g. just failover to "Last Changed Date: unknown; not SVN" if SVN-REVISION is missing?

I am wondering if there is any plan to clean up this situation because it seems like a stumbling block for users who might want to contribute to R.

Thanks,

Frederick

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: best way to build from Git

Gábor Csárdi
Hi Frederick,

I know some non R-core people use this workflow to keep local patches
in git branches:
https://bookdown.org/lionel/contributing/

Best,
Gabor

On Thu, Dec 26, 2019 at 1:37 AM <[hidden email]> wrote:

>
> Dear R-devel,
>
> I checked out a recent copy of R via Subversion and made a few changes to the code. I wanted to commit them locally to my repo, just to stay organized and keep them separate from other changes I plan to make. However, I was not able to commit them because I don't think SVN allows this?
>
> Plan B, check out a Git mirror of R, https://github.com/wch/r-source/
>
> It has some instructions for building: https://github.com/wch/r-source/wiki
>
> However, there are some complicated workarounds due to a commit from 2013 (https://github.com/wch/r-source/commit/4f13e5325dfbcb9fc8f55fc6027af9ae9c7750a3). And it only seems to succeed if the build directory is the same as the source directory.
>
> Makefile.in has some odd stuff that might be related, like why is it checking for "$(srcdir)/doc/FAQ" rather than "$(top_builddir)/doc/FAQ"? Conversely, there is a check for "$(top_builddir)/.git" which I would expect to be in "$(srcdir)/.git" instead. And it runs "git svn info" which seems to take indefinitely long - I am not using "git svn", I'm just trying to build from a regular git repo. This is difficult to debug because it is inside an @if, so "make -n" still runs it. Also there are no comments to explain what e.g. the "svnonly" target is doing, or anything else.
>
> After some time I still have little idea why it is necessary for the build system to depend on the revision control system. Clearly we are trying to do something fancy, but why not e.g. just failover to "Last Changed Date: unknown; not SVN" if SVN-REVISION is missing?
>
> I am wondering if there is any plan to clean up this situation because it seems like a stumbling block for users who might want to contribute to R.
>
> Thanks,
>
> Frederick
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: best way to build from Git

Duncan Murdoch-2
In reply to this post by frederik-2
On 25/12/2019 8:37 p.m., [hidden email] wrote:
> Dear R-devel,
>
> I checked out a recent copy of R via Subversion and made a few changes to the code. I wanted to commit them locally to my repo, just to stay organized and keep them separate from other changes I plan to make. However, I was not able to commit them because I don't think SVN allows this?

What I used to do in that situation was to run "svn diff >patchfile"
(maybe with some options, I forget), then save the patch locally.  "svn
patch" can usually apply the patch appropriately after the working copy
has been updated:  it understands the svn revision numbers recorded in
the patch.  (Conflicts cause trouble, but no more than any other time.)
This doesn't work so well if you are adding or moving files.

The patch produced in this way is also suitable for including in a bug
report on bugs.r-project.org.

Duncan Murdoch

>
> Plan B, check out a Git mirror of R, https://github.com/wch/r-source/
>
> It has some instructions for building: https://github.com/wch/r-source/wiki
>
> However, there are some complicated workarounds due to a commit from 2013 (https://github.com/wch/r-source/commit/4f13e5325dfbcb9fc8f55fc6027af9ae9c7750a3). And it only seems to succeed if the build directory is the same as the source directory.
>
> Makefile.in has some odd stuff that might be related, like why is it checking for "$(srcdir)/doc/FAQ" rather than "$(top_builddir)/doc/FAQ"? Conversely, there is a check for "$(top_builddir)/.git" which I would expect to be in "$(srcdir)/.git" instead. And it runs "git svn info" which seems to take indefinitely long - I am not using "git svn", I'm just trying to build from a regular git repo. This is difficult to debug because it is inside an @if, so "make -n" still runs it. Also there are no comments to explain what e.g. the "svnonly" target is doing, or anything else.
>
> After some time I still have little idea why it is necessary for the build system to depend on the revision control system. Clearly we are trying to do something fancy, but why not e.g. just failover to "Last Changed Date: unknown; not SVN" if SVN-REVISION is missing?
>
> I am wondering if there is any plan to clean up this situation because it seems like a stumbling block for users who might want to contribute to R.

I

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: best way to build from Git

Martin Maechler
In reply to this post by Gábor Csárdi
>>>>> Gábor Csárdi
>>>>>     on Thu, 26 Dec 2019 08:23:10 +0000 writes:

    > Hi Frederick, I know some non R-core people use this
    > workflow to keep local patches in git branches:
    > https://bookdown.org/lionel/contributing/

    > Best, Gabor

Thank you, Gabor, and notably, Lionel, for providing the extras.

As Frederik notes / ask as well about it:
R-core people typically never build "inside the source tree"
(not "polluting" the source tree by built stuff)
but rather in separate build directory, as I'm pretty sure we've
recommended for years.

It would probably really be useful, if Lionel (or someone else)
updated his nice write-up and auxiliary {script / make} - files
so things work too when build with "src-dir != build-dir".

    > On Thu, Dec 26, 2019 at 1:37 AM <[hidden email]> wrote:
    >>
    >> Dear R-devel,
    >>
    >> I checked out a recent copy of R via Subversion and made
    >> a few changes to the code. I wanted to commit them
    >> locally to my repo, just to stay organized and keep them
    >> separate from other changes I plan to make. However, I
    >> was not able to commit them because I don't think SVN
    >> allows this?
    >>
    >> Plan B, check out a Git mirror of R,
    >> https://github.com/wch/r-source/
    >>
    >> It has some instructions for building:
    >> https://github.com/wch/r-source/wiki
    >>
    >> However, there are some complicated workarounds due to a
    >> commit from 2013
    >> (https://github.com/wch/r-source/commit/4f13e5325dfbcb9fc8f55fc6027af9ae9c7750a3).

    >> And it only seems to succeed if the build directory is the
    >> same as the source directory.

(I have not tried, as I as R-corer  do work with svn, but) If
this is true, then that is a pity indeed.
I have stopped of *ever* building in the source directory many
years ago, and I'm pretty sure that is the case for the vast
majority of R-corers.

    >> Makefile.in has some odd stuff that might be related,
    >> like why is it checking for "$(srcdir)/doc/FAQ" rather
    >> than "$(top_builddir)/doc/FAQ"? Conversely, there is a
    >> check for "$(top_builddir)/.git" which I would expect to
    >> be in "$(srcdir)/.git" instead. And it runs "git svn
    >> info" which seems to take indefinitely long - I am not
    >> using "git svn", I'm just trying to build from a regular
    >> git repo. This is difficult to debug because it is inside
    >> an @if, so "make -n" still runs it. Also there are no
    >> comments to explain what e.g. the "svnonly" target is
    >> doing, or anything else.

"svnonly" is mostly to distinguish an 'svn checkout' (which is
what all of R core work with) from the case of
(pre)released R, i.e., a source tarball which of course at this point in time is
unpacked into a source directory tree and to make sure that the
tarball does have a valid svn revision, via the SVN-REVISION
file (which now *is* in all tarballs as you can see).


    >> After some time I still have little idea why it is
    >> necessary for the build system to depend on the revision
    >> control system. Clearly we are trying to do something
    >> fancy, but why not e.g. just failover to "Last Changed
    >> Date: unknown; not SVN" if SVN-REVISION is missing?

almost surely because that will fail later: It is needed to
build the exact "R version":

> str(version)
List of 14
 $ platform      : chr "x86_64-pc-linux-gnu"
 $ arch          : chr "x86_64"
 $ os            : chr "linux-gnu"
 $ system        : chr "x86_64, linux-gnu"
 $ status        : chr ""
 $ major         : chr "3"
 $ minor         : chr "6.2"
 $ year          : chr "2019"
 $ month         : chr "12"
 $ day           : chr "12"
 $ svn rev       : chr "77560"
 $ language      : chr "R"
 $ version.string: chr "R version 3.6.2 (2019-12-12)"
 $ nickname      : chr "Dark and Stormy Night"
 - attr(*, "class")= chr "simple.list"
>


    >> I am wondering if there is any plan to clean up this
    >> situation because it seems like a stumbling block for
    >> users who might want to contribute to R.

    >> Thanks,
    >>
    >> Frederick

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: best way to build from Git

Gábor Csárdi
On Thu, Dec 26, 2019 at 12:39 PM Martin Maechler
<[hidden email]> wrote:
[...]
> As Frederik notes / ask as well about it:
> R-core people typically never build "inside the source tree"
> (not "polluting" the source tree by built stuff)
> but rather in separate build directory, as I'm pretty sure we've
> recommended for years.
>
> It would probably really be useful, if Lionel (or someone else)
> updated his nice write-up and auxiliary {script / make} - files
> so things work too when build with "src-dir != build-dir".

I am not sure what you mean, what is wrong with this build directory?
https://bookdown.org/lionel/contributing/building-r.html#sec:build

Gabor

[...]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: best way to build from Git

Duncan Murdoch-2
In reply to this post by Martin Maechler
On 26/12/2019 7:38 a.m., Martin Maechler wrote:

>>>>>> Gábor Csárdi
>>>>>>      on Thu, 26 Dec 2019 08:23:10 +0000 writes:
>
>      > Hi Frederick, I know some non R-core people use this
>      > workflow to keep local patches in git branches:
>      > https://bookdown.org/lionel/contributing/
>
>      > Best, Gabor
>
> Thank you, Gabor, and notably, Lionel, for providing the extras.
>
> As Frederik notes / ask as well about it:
> R-core people typically never build "inside the source tree"
> (not "polluting" the source tree by built stuff)
> but rather in separate build directory, as I'm pretty sure we've
> recommended for years.
>
> It would probably really be useful, if Lionel (or someone else)
> updated his nice write-up and auxiliary {script / make} - files
> so things work too when build with "src-dir != build-dir".
>
>      > On Thu, Dec 26, 2019 at 1:37 AM <[hidden email]> wrote:
>      >>
>      >> Dear R-devel,
>      >>
>      >> I checked out a recent copy of R via Subversion and made
>      >> a few changes to the code. I wanted to commit them
>      >> locally to my repo, just to stay organized and keep them
>      >> separate from other changes I plan to make. However, I
>      >> was not able to commit them because I don't think SVN
>      >> allows this?
>      >>
>      >> Plan B, check out a Git mirror of R,
>      >> https://github.com/wch/r-source/
>      >>
>      >> It has some instructions for building:
>      >> https://github.com/wch/r-source/wiki
>      >>
>      >> However, there are some complicated workarounds due to a
>      >> commit from 2013
>      >> (https://github.com/wch/r-source/commit/4f13e5325dfbcb9fc8f55fc6027af9ae9c7750a3).
>
>      >> And it only seems to succeed if the build directory is the
>      >> same as the source directory.
>
> (I have not tried, as I as R-corer  do work with svn, but) If
> this is true, then that is a pity indeed.
> I have stopped of *ever* building in the source directory many
> years ago, and I'm pretty sure that is the case for the vast
> majority of R-corers.

When I was building for Windows, the workflow required the build to be
in the source tree.  Windows systems weren't assumed to be able to run
configure, so all the work it does was done manually for the Rtools tools.

Duncan Murdoch

>
>      >> Makefile.in has some odd stuff that might be related,
>      >> like why is it checking for "$(srcdir)/doc/FAQ" rather
>      >> than "$(top_builddir)/doc/FAQ"? Conversely, there is a
>      >> check for "$(top_builddir)/.git" which I would expect to
>      >> be in "$(srcdir)/.git" instead. And it runs "git svn
>      >> info" which seems to take indefinitely long - I am not
>      >> using "git svn", I'm just trying to build from a regular
>      >> git repo. This is difficult to debug because it is inside
>      >> an @if, so "make -n" still runs it. Also there are no
>      >> comments to explain what e.g. the "svnonly" target is
>      >> doing, or anything else.
>
> "svnonly" is mostly to distinguish an 'svn checkout' (which is
> what all of R core work with) from the case of
> (pre)released R, i.e., a source tarball which of course at this point in time is
> unpacked into a source directory tree and to make sure that the
> tarball does have a valid svn revision, via the SVN-REVISION
> file (which now *is* in all tarballs as you can see).
>
>
>      >> After some time I still have little idea why it is
>      >> necessary for the build system to depend on the revision
>      >> control system. Clearly we are trying to do something
>      >> fancy, but why not e.g. just failover to "Last Changed
>      >> Date: unknown; not SVN" if SVN-REVISION is missing?
>
> almost surely because that will fail later: It is needed to
> build the exact "R version":
>
>> str(version)
> List of 14
>   $ platform      : chr "x86_64-pc-linux-gnu"
>   $ arch          : chr "x86_64"
>   $ os            : chr "linux-gnu"
>   $ system        : chr "x86_64, linux-gnu"
>   $ status        : chr ""
>   $ major         : chr "3"
>   $ minor         : chr "6.2"
>   $ year          : chr "2019"
>   $ month         : chr "12"
>   $ day           : chr "12"
>   $ svn rev       : chr "77560"
>   $ language      : chr "R"
>   $ version.string: chr "R version 3.6.2 (2019-12-12)"
>   $ nickname      : chr "Dark and Stormy Night"
>   - attr(*, "class")= chr "simple.list"
>>
>
>
>      >> I am wondering if there is any plan to clean up this
>      >> situation because it seems like a stumbling block for
>      >> users who might want to contribute to R.
>
>      >> Thanks,
>      >>
>      >> Frederick
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: best way to build from Git

frederik-2
In reply to this post by Gábor Csárdi
Thank you all for your replies.

On Thu, Dec 26, 2019 at 01:06:48PM +0000, Gábor Csárdi wrote:
>On Thu, Dec 26, 2019 at 12:39 PM Martin Maechler
>> ...
>> It would probably really be useful, if Lionel (or someone else)
>> updated his nice write-up and auxiliary {script / make} - files
>> so things work too when build with "src-dir != build-dir".
>
>I am not sure what you mean, what is wrong with this build directory?
>https://bookdown.org/lionel/contributing/building-r.html#sec:build

Yes it seems that Lionel's instructions successfully build a Git repo
containing R into a separate build directory. Thank you Lionel.

I've adapted these instructions, together with Lionel's GNUmakefile
and make-svn-revision script, to suit my (Linux-based) environment. I
put it in a little GitHub repository. Currently the following commands
work for me to download and install R in my home directory:

     git clone https://github.com/navarum/tweaks
     PREFIX=~/.local ./tweaks/r/BUILD install

For now it applies one example patch on top of a fixed recent commit
to the upstream repo https://github.com/wch/r-source/ . I noticed that
the first commit hash I selected from 'git log' was not stable, so I'm
not sure if this script will work tomorrow or next week. However, I
thought I ought to share it in case others find it useful. I tried to
make the (Bash) code concise enough that it would be possible for
others to see what it is doing and adapt it to maintaining their own
patch series on top of some version of R. I've been using this tool to
maintain custom versions of other software packages as well, so there
may be a little "too much information" in there - but someone could
obviously fork it and keep just what they need. The system tries to be
somewhat intelligent about remembering whether configure was run,
checking if the patches need to be reapplied, not overwriting any
changes I might have made to the cloned r-source git repo, and so on.

Just for the record, my original question on this thread was about the
reasons for not having R's default build system work with different
revision control systems, as well as some apparent typos in the
Makefile that comes with R.

Thank you,

Frederick

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel