Notes on building a gcc toolchain for Rtools (but not multilib)

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Notes on building a gcc toolchain for Rtools (but not multilib)

Hsiu-Khuern Tang-2
Hi,

[This is a follow-up to the "New version of Rtools for Windows" thread
in January, but I just subscribed and don't know how to reply to an
old thread -- my apologies.]

I was able to use the nuwen distro to build a gcc 4.9.2 toolchain and
use it to build the latest R-patched with it.

Below are some notes about what I did; I hope they will be useful for
keeping Rtools up-to-date.


Note:

- This is 64-bit only; I tried but was unable to create a multilib toolchain

- I did not run any tests on the resulting R binary, other than
starting R and running some basic commands

- I don't necessarily know what I'm doing!


Outline of steps:

- The (nicely done!) nuwen website by Stephan Lavavej has made
available a MinGW distro and the scripts used to create it.

  + However, the gcc toolchain there is built with --disable-lib32 (so
no multilib) and with --disable-gomp (the default)

  + Moreover, the pthreads library is not included in the distro

  + I believe building R requires GOMP and pthreads, hence I tried to
modify the scripts to add these

- Installing your own toolchain

  + Read the instructions in the section "How To Build Your Own Distro"

  + You don't have to rebuild everything in components-12.2.7z: you
only need the original binutils-2.25.7z and your own build of
mingw-w64+gcc.7z to replace Rtools's gcc

  + You will need to run a modified version of Stephan's
mingw-w64+gcc.sh script.  Besides the gcc source code, you will also
need to download the pthreads-w32 source code from
https://www.sourceware.org/pthreads-win32/

  + Here are the changes I made to the mingw-w64+gcc.sh script:

--------------------------------------------------

diff --git a/mingw-w64+gcc.sh b/mingw-w64+gcc.sh
index 2402ffc..fd44e76 100644
--- a/mingw-w64+gcc.sh
+++ b/mingw-w64+gcc.sh
@@ -8,6 +8,7 @@ source 0_append_distro_path.sh
 7z x '-oC:\Temp\gcc' gmp-6.0.0a.tar > NUL || fail_with gmp-6.0.0a.tar
- EPIC FAIL
 7z x '-oC:\Temp\gcc' mpfr-3.1.2.tar > NUL || fail_with mpfr-3.1.2.tar
- EPIC FAIL
 7z x '-oC:\Temp\gcc' mpc-1.0.2.tar > NUL || fail_with mpc-1.0.2.tar - EPIC FAIL
+7z x '-oC:\Temp\gcc' pthreads-w32-2-9-1-release.tar > NUL ||
fail_with pthreads-w32-2-9-1-release.tar - EPIC FAIL

 patch -Z -d /c/temp/gcc/mpfr-3.1.2 -p1 < mpfr.patch

@@ -25,6 +26,14 @@ make all install "CFLAGS=-s -O3" || fail_with
mingw-w64 make - EPIC FAIL
 cd /c/temp/gcc
 rm -rf build src

+# Build pthreads-w32.
+cd pthreads-w32-2-9-1-release
+make clean GC
+
+cp libpthreadGC2.a ../dest/x86_64-w64-mingw32/lib/libpthread.a
+cp pthread.h sched.h semaphore.h ../dest/x86_64-w64-mingw32/include
+cp pthreadGC2.dll $X_DISTRO_ROOT/bin/
+
 # Prepare to build gcc - set up the in-tree builds of gmp, mpfr, and mpc.
 mv gcc-4.9.2 src
 mv gmp-6.0.0 src/gmp
@@ -40,7 +49,7 @@ cp -r dest/x86_64-w64-mingw32/include
src/gcc/winsup/mingw/include
 # Configure.
 mkdir build
 cd build
-../src/configure --enable-languages=c,c++ --build=x86_64-w64-mingw32
--host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
--disable-multilib --prefix=/c/temp/gcc/dest
--with-sysroot=/c/temp/gcc/dest --disable-libstdcxx-pch --disable-lto
--disable-nls --disable-shared --disable-win32-registry
--enable-checking=release --with-tune=haswell || fail_with gcc
configure - EPIC FAIL
+../src/configure --enable-languages=c,c++,fortran
--build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
--target=x86_64-w64-mingw32 --disable-multilib
--prefix=/c/temp/gcc/dest --with-sysroot=/c/temp/gcc/dest
--disable-libstdcxx-pch --disable-lto --disable-nls --disable-shared
--disable-win32-registry --enable-libgomp --enable-checking=release
--with-tune=haswell || fail_with gcc configure - EPIC FAIL

 # --enable-languages=c,c++        : I want C and C++ only.
 # --build=x86_64-w64-mingw32      : I want a native compiler.

--------------------------------------------------

  + After running this script, you'll get your own mingw-w64+gcc.7z.

    - You'll need some patience -- the full 3-stage bootstrap of gcc
took 10 hours for me

    - You can replace the "make bootstrap" in the script by "make
bootstrap2" to omit the last stage.

  + Install your toolchain:

    - If you haven't done so, install Rtools.  We will still use the
command line utilities in C:\Rtools\bin

    - Make a new directory, e.g., C:\Rtools\nuwen and unpack
binutils-2.25.7z and mingw-w64+gcc.7z there

    - Put the above directory in your PATH, in front of any other
toolchain locations such as C:\Rtools\gcc-4.6.3\bin (it may be better
to remove the latter from your PATH)

- Now for the installation of R:

  + Get the latest R-patched sources (rather than 3.1.2, because
Duncan (thanks!) have added some useful changes)

  + Make sure you have the prerequisites for building R (see the R
Installation and Administration Manual)

    - In particular, the source files for the recommended packages,
the support files for Tcl, and the "extsoft" headers and libraries

  + In src/gnuwin32, copy MkRules.dist to MkRules.local and apply the
following patch:

--------------------------------------------------

--- MkRules.local 2015-03-02 13:57:38.601903500 -0800
+++ MkRules.local 2015-03-06 10:43:44.708581800 -0800
@@ -52,6 +52,7 @@
 # BINPREF =
 # prefix for 64-bit: path or x86_64-w64-mingw32-
 # BINPREF64 = x86_64-w64-mingw32-
+BINPREF64 =

 # Others use a -m64 or -m32 option to select architectures
 # M_ARCH =
@@ -64,6 +65,7 @@

 # 32- or 64-bit Windows?
 # WIN = 32
+WIN = 64

 # The gcc 4.9.2 64 bit toolchain is set up for the 'medium code'
model and needs
 # to remove the .refptr entries from the exports list; this is the default
@@ -135,12 +137,12 @@
 # Full paths of extra DLLs that need to be shipped
 # e.g
 # DLLs32 = c:/R/bin/pthreadGC2-w32.dll
-# DLLs64 = c:/R/bin64/pthreadGC2-w64.dll
+DLLs64 = c:/MinGW/bin/pthreadGC2.dll
 # DLLs32 =
 # DLLs64 =

 # Define this to 1 if using the gcc 4.9.2 toolchain with dynamic linking
-# COPY_RUNTIME_DLLS =
+COPY_RUNTIME_DLLS = 1

 ## ====== configuration macros for building MSI installer ===========

--------------------------------------------------

  + (I don't know whether the DLLs64 and COPY_RUNTIME_DLLS changes
above are necessary or not.  C:/MinGW/bin is where I put the pthreads
DLL that was built earlier)

  + Run "make all recommended".  If this works, you should have a
working R, built using your new toolchain!



Additional notes:

- I tried using my new R to install Rcpp from source, but this failed
because the R build scripts was not able to determine the right set of
symbols to be exported in the Rcpp DLL.  To solve this, edit the file
etc/x64/Makeconf under your R source tree, replacing

NM_FILTER = | sed -e '/.refptr./d'

by

NM_FILTER = | sed -e '/.refptr./d; /\.weak\./d'


Hope this helps,

- Hsiu-Khuern

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Avraham Adler
On Sun, Mar 8, 2015 at 10:02 PM, Hsiu-Khuern Tang <[hidden email]> wrote:

> Hi,
>
> [This is a follow-up to the "New version of Rtools for Windows" thread
> in January, but I just subscribed and don't know how to reply to an
> old thread -- my apologies.]
>
> I was able to use the nuwen distro to build a gcc 4.9.2 toolchain and
> use it to build the latest R-patched with it.
>
> Below are some notes about what I did; I hope they will be useful for
> keeping Rtools up-to-date.
>
>
> Note:
>
> - This is 64-bit only; I tried but was unable to create a multilib toolchain
>
> - I did not run any tests on the resulting R binary, other than
> starting R and running some basic commands
>
> - I don't necessarily know what I'm doing!
>

Thank you for the  update, Hsiu-Khuern. Can you check how many digits
scientific notation shows? I was able to build a 64bit version of R
using the mingw64 4.8.4 distro, but ran into trouble with compat.c,
and the only work-around I found ended breaking R's defualt two digit
notation and used Windows's three-digit notation; it seems as if you
did not have this problem. Also, do you plan on running 'make check'
eventually on your build?

Avi

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Hsiu-Khuern Tang-2
On Sun, Mar 8, 2015 at 8:33 PM, Avraham Adler <[hidden email]> wrote:

>
> Thank you for the  update, Hsiu-Khuern. Can you check how many digits
> scientific notation shows? I was able to build a 64bit version of R
> using the mingw64 4.8.4 distro, but ran into trouble with compat.c,
> and the only work-around I found ended breaking R's defualt two digit
> notation and used Windows's three-digit notation; it seems as if you
> did not have this problem.

I get notation like "1e-09".  I think (but am not sure) that there
were some changes in mingw's stdio.h that got rid of the errors about
redefined *sprintf functions.

> Also, do you plan on running 'make check'
> eventually on your build?

I don't have any plans to do that, mainly for fear that I would find
some problems and be sucked down the rabbit hole of trying to solve
them!  Hoping someone else more knowledgeable will take the plunge :)

>
> Avi
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Avraham Adler
Spurred by Hsiu-Khuern's post, I was able to compile
R-devel_2015-03-08 using the already compiled distribution at
<http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.9.2/threads-win32/seh/x86_64-4.9.2-release-win32-seh-rt_v3-rev1.7z/download>This
GCC 4.9.2 using seh error handling and windows threads, so as per this
note <https://stat.ethz.ch/pipermail/r-devel/2015-March/070782.html>,
I left out ICU. I also did not use curl for this build, I applied
Hsiu-Khuern's suggestion to the NM filtering, and I left leaving
Rtools\bin to handle utilities like tar.

I built all, cairodevices, recommended, vignettes, manuals, and
rinstaller, and then ran make check-all. The only .out.fail file that
appeared was the one for "internet.R" where I got a "connect: No
error". That was fixed inside R by using setInternet2(). This has not
happened for me before; has the internet handling in r-devel been
changed? Is there a way to have that properly set in the Makefile?
Skimming through the logs of the make check-all, some vignettes failed
to generate as the packages they depend on could not be loaded. If
there was a way to have the proper internet connectivity set when
being compiled, it appears this would have passed make check-all
completely.

This build was also compiled against OpenBLAS 2.13 and a bunch of
optimizations (specifically -O3 -march=native -mfpmath=sse -msse2avx
-mavx256-split-unaligned-load -mavx256-split-unaligned-store
-mvzeroupper -std=gnu++11 -pipe) and it still passed.

While I uncommented OPENMP and PTHREAD in Mkrules, I have not
explicitly checked them (unless that is handled in check-all). Also, I
only built 64 bit, I did not try to simultaneously build a 32-bit
version. However, being that this distribution is already compiled and
exists, it may be easier to use than having to custom build off of the
neuwen distro, and perhaps someone with more skill can try the 32 bit
version and even getting curl to work (which may sidestep the internet
problem I had above).

Thank you,

Avi

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
In reply to this post by Hsiu-Khuern Tang-2
On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> Hi,
>
> [This is a follow-up to the "New version of Rtools for Windows" thread
> in January, but I just subscribed and don't know how to reply to an
> old thread -- my apologies.]

I am planning to put a new Rtools online today that uses a different
build of gcc 4.9.2.  I will be concentrating on getting it to work with
all the external libraries before the 3.2.0 release next month.  I'm not
planning to try to get it to work with R-patched, and I expect it won't:
 I needed to make a number of patches to R-devel for compatibility.

It is also not multilib.  It uses static linking of libraries, and SJLJ
exception handling in both 32 and 64 bit builds -- this is necessary for
compatibility with existing compiles of external libraries, but may not
be ideal in 64 bits.  However, I believe using SEH will mean static
linking is impossible, and that leads to problems distributing run-time
DLLs, and I'd rather not worry about those.

Testing of the new Rtools will be appreciated.  I expect the R-devel
builds will start using it tomorrow or the next day.

Duncan Murdoch

>
> I was able to use the nuwen distro to build a gcc 4.9.2 toolchain and
> use it to build the latest R-patched with it.
>
> Below are some notes about what I did; I hope they will be useful for
> keeping Rtools up-to-date.
>
>
> Note:
>
> - This is 64-bit only; I tried but was unable to create a multilib toolchain
>
> - I did not run any tests on the resulting R binary, other than
> starting R and running some basic commands
>
> - I don't necessarily know what I'm doing!
>
>
> Outline of steps:
>
> - The (nicely done!) nuwen website by Stephan Lavavej has made
> available a MinGW distro and the scripts used to create it.
>
>   + However, the gcc toolchain there is built with --disable-lib32 (so
> no multilib) and with --disable-gomp (the default)
>
>   + Moreover, the pthreads library is not included in the distro
>
>   + I believe building R requires GOMP and pthreads, hence I tried to
> modify the scripts to add these
>
> - Installing your own toolchain
>
>   + Read the instructions in the section "How To Build Your Own Distro"
>
>   + You don't have to rebuild everything in components-12.2.7z: you
> only need the original binutils-2.25.7z and your own build of
> mingw-w64+gcc.7z to replace Rtools's gcc
>
>   + You will need to run a modified version of Stephan's
> mingw-w64+gcc.sh script.  Besides the gcc source code, you will also
> need to download the pthreads-w32 source code from
> https://www.sourceware.org/pthreads-win32/
>
>   + Here are the changes I made to the mingw-w64+gcc.sh script:
>
> --------------------------------------------------
>
> diff --git a/mingw-w64+gcc.sh b/mingw-w64+gcc.sh
> index 2402ffc..fd44e76 100644
> --- a/mingw-w64+gcc.sh
> +++ b/mingw-w64+gcc.sh
> @@ -8,6 +8,7 @@ source 0_append_distro_path.sh
>  7z x '-oC:\Temp\gcc' gmp-6.0.0a.tar > NUL || fail_with gmp-6.0.0a.tar
> - EPIC FAIL
>  7z x '-oC:\Temp\gcc' mpfr-3.1.2.tar > NUL || fail_with mpfr-3.1.2.tar
> - EPIC FAIL
>  7z x '-oC:\Temp\gcc' mpc-1.0.2.tar > NUL || fail_with mpc-1.0.2.tar - EPIC FAIL
> +7z x '-oC:\Temp\gcc' pthreads-w32-2-9-1-release.tar > NUL ||
> fail_with pthreads-w32-2-9-1-release.tar - EPIC FAIL
>
>  patch -Z -d /c/temp/gcc/mpfr-3.1.2 -p1 < mpfr.patch
>
> @@ -25,6 +26,14 @@ make all install "CFLAGS=-s -O3" || fail_with
> mingw-w64 make - EPIC FAIL
>  cd /c/temp/gcc
>  rm -rf build src
>
> +# Build pthreads-w32.
> +cd pthreads-w32-2-9-1-release
> +make clean GC
> +
> +cp libpthreadGC2.a ../dest/x86_64-w64-mingw32/lib/libpthread.a
> +cp pthread.h sched.h semaphore.h ../dest/x86_64-w64-mingw32/include
> +cp pthreadGC2.dll $X_DISTRO_ROOT/bin/
> +
>  # Prepare to build gcc - set up the in-tree builds of gmp, mpfr, and mpc.
>  mv gcc-4.9.2 src
>  mv gmp-6.0.0 src/gmp
> @@ -40,7 +49,7 @@ cp -r dest/x86_64-w64-mingw32/include
> src/gcc/winsup/mingw/include
>  # Configure.
>  mkdir build
>  cd build
> -../src/configure --enable-languages=c,c++ --build=x86_64-w64-mingw32
> --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
> --disable-multilib --prefix=/c/temp/gcc/dest
> --with-sysroot=/c/temp/gcc/dest --disable-libstdcxx-pch --disable-lto
> --disable-nls --disable-shared --disable-win32-registry
> --enable-checking=release --with-tune=haswell || fail_with gcc
> configure - EPIC FAIL
> +../src/configure --enable-languages=c,c++,fortran
> --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
> --target=x86_64-w64-mingw32 --disable-multilib
> --prefix=/c/temp/gcc/dest --with-sysroot=/c/temp/gcc/dest
> --disable-libstdcxx-pch --disable-lto --disable-nls --disable-shared
> --disable-win32-registry --enable-libgomp --enable-checking=release
> --with-tune=haswell || fail_with gcc configure - EPIC FAIL
>
>  # --enable-languages=c,c++        : I want C and C++ only.
>  # --build=x86_64-w64-mingw32      : I want a native compiler.
>
> --------------------------------------------------
>
>   + After running this script, you'll get your own mingw-w64+gcc.7z.
>
>     - You'll need some patience -- the full 3-stage bootstrap of gcc
> took 10 hours for me
>
>     - You can replace the "make bootstrap" in the script by "make
> bootstrap2" to omit the last stage.
>
>   + Install your toolchain:
>
>     - If you haven't done so, install Rtools.  We will still use the
> command line utilities in C:\Rtools\bin
>
>     - Make a new directory, e.g., C:\Rtools\nuwen and unpack
> binutils-2.25.7z and mingw-w64+gcc.7z there
>
>     - Put the above directory in your PATH, in front of any other
> toolchain locations such as C:\Rtools\gcc-4.6.3\bin (it may be better
> to remove the latter from your PATH)
>
> - Now for the installation of R:
>
>   + Get the latest R-patched sources (rather than 3.1.2, because
> Duncan (thanks!) have added some useful changes)
>
>   + Make sure you have the prerequisites for building R (see the R
> Installation and Administration Manual)
>
>     - In particular, the source files for the recommended packages,
> the support files for Tcl, and the "extsoft" headers and libraries
>
>   + In src/gnuwin32, copy MkRules.dist to MkRules.local and apply the
> following patch:
>
> --------------------------------------------------
>
> --- MkRules.local 2015-03-02 13:57:38.601903500 -0800
> +++ MkRules.local 2015-03-06 10:43:44.708581800 -0800
> @@ -52,6 +52,7 @@
>  # BINPREF =
>  # prefix for 64-bit: path or x86_64-w64-mingw32-
>  # BINPREF64 = x86_64-w64-mingw32-
> +BINPREF64 =
>
>  # Others use a -m64 or -m32 option to select architectures
>  # M_ARCH =
> @@ -64,6 +65,7 @@
>
>  # 32- or 64-bit Windows?
>  # WIN = 32
> +WIN = 64
>
>  # The gcc 4.9.2 64 bit toolchain is set up for the 'medium code'
> model and needs
>  # to remove the .refptr entries from the exports list; this is the default
> @@ -135,12 +137,12 @@
>  # Full paths of extra DLLs that need to be shipped
>  # e.g
>  # DLLs32 = c:/R/bin/pthreadGC2-w32.dll
> -# DLLs64 = c:/R/bin64/pthreadGC2-w64.dll
> +DLLs64 = c:/MinGW/bin/pthreadGC2.dll
>  # DLLs32 =
>  # DLLs64 =
>
>  # Define this to 1 if using the gcc 4.9.2 toolchain with dynamic linking
> -# COPY_RUNTIME_DLLS =
> +COPY_RUNTIME_DLLS = 1
>
>  ## ====== configuration macros for building MSI installer ===========
>
> --------------------------------------------------
>
>   + (I don't know whether the DLLs64 and COPY_RUNTIME_DLLS changes
> above are necessary or not.  C:/MinGW/bin is where I put the pthreads
> DLL that was built earlier)
>
>   + Run "make all recommended".  If this works, you should have a
> working R, built using your new toolchain!
>
>
>
> Additional notes:
>
> - I tried using my new R to install Rcpp from source, but this failed
> because the R build scripts was not able to determine the right set of
> symbols to be exported in the Rcpp DLL.  To solve this, edit the file
> etc/x64/Makeconf under your R source tree, replacing
>
> NM_FILTER = | sed -e '/.refptr./d'
>
> by
>
> NM_FILTER = | sed -e '/.refptr./d; /\.weak\./d'
>
>
> Hope this helps,
>
> - Hsiu-Khuern
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Hsiu-Khuern Tang-2
On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]> wrote:

> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>> Hi,
>>
>> [This is a follow-up to the "New version of Rtools for Windows" thread
>> in January, but I just subscribed and don't know how to reply to an
>> old thread -- my apologies.]
>
> I am planning to put a new Rtools online today that uses a different
> build of gcc 4.9.2.  I will be concentrating on getting it to work with
> all the external libraries before the 3.2.0 release next month.  I'm not
> planning to try to get it to work with R-patched, and I expect it won't:
>  I needed to make a number of patches to R-devel for compatibility.

I also worked off R-devel (I said wrongly that it was R-patched in my
original post) and benefited from your compatibility changes.

I look forward to the new Rtools and will test it by compiling some packages.

- Hsiu-Khuern

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Avraham Adler
In reply to this post by Duncan Murdoch-2
On Mon, Mar 9, 2015 at 6:50 AM, Duncan Murdoch <[hidden email]> wrote:

> It is also not multilib.  It uses static linking of libraries, and SJLJ
> exception handling in both 32 and 64 bit builds -- this is necessary for
> compatibility with existing compiles of external libraries, but may not
> be ideal in 64 bits.  However, I believe using SEH will mean static
> linking is impossible, and that leads to problems distributing run-time
> DLLs, and I'd rather not worry about those.

That is great news, Dr. Murdoch.

I know that ICU, curl, and the files in goodies (lzma, jpeg ,etc.) are
now external libraries necessary for compiling R. Is that all, or are
there (many) others of which I am unaware? I understand that given the
release date of 3.2.0, the focus needs to be on having a functional
toolset and all that entails, and if that means sjlj is used for now,
that is perfectly reasonable.

However, if the number of such external libraries is manageable, would
it make sense to recompile /them/ using an seh toolchain to take
advantage of the increased performance?

Thank you,

Avi

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Avraham Adler
In reply to this post by Hsiu-Khuern Tang-2
On Mon, Mar 9, 2015 at 11:07 AM, Hsiu-Khuern Tang <[hidden email]> wrote:
> I also worked off R-devel (I said wrongly that it was R-patched in my
> original post) and benefited from your compatibility changes.

Hsiu-Khuern, using your suggest filter in NM, I was able to build Rcpp
in the R_devel compiled with Gcc 4.9.2/win32/seh last night, and I saw
around a 10%-15% speedup in Rcpp performance (and R performance in
general) from the 3.1.2 version I compiled with Rtool 4.6.3. That may
be related to inherent optimizations in 4.9.2 vs 4.6.3, it may be
related to seh vs sjlj, or it may be related to passing gnu++11
instead of gnu++0x, I don't know. I do know it was the same code on
the same machine, for what it is worth.

Thank you,

Avi

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
In reply to this post by Hsiu-Khuern Tang-2
On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:

> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]> wrote:
> > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> >> Hi,
> >>
> >> [This is a follow-up to the "New version of Rtools for Windows" thread
> >> in January, but I just subscribed and don't know how to reply to an
> >> old thread -- my apologies.]
> >
> > I am planning to put a new Rtools online today that uses a different
> > build of gcc 4.9.2.  I will be concentrating on getting it to work with
> > all the external libraries before the 3.2.0 release next month.  I'm not
> > planning to try to get it to work with R-patched, and I expect it won't:
> >  I needed to make a number of patches to R-devel for compatibility.
>
> I also worked off R-devel (I said wrongly that it was R-patched in my
> original post) and benefited from your compatibility changes.
>
> I look forward to the new Rtools and will test it by compiling some packages.

It's now on the main site at CRAN, and should propagate to the mirrors
reasonably quickly.  I'm hoping that tomorrow's R-devel build will use
it, but there may be some last minute problems.

Duncan Murdoch

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
On 09/03/2015 1:40 PM, Duncan Murdoch wrote:

> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> > On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]> wrote:
> > > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> > >> Hi,
> > >>
> > >> [This is a follow-up to the "New version of Rtools for Windows" thread
> > >> in January, but I just subscribed and don't know how to reply to an
> > >> old thread -- my apologies.]
> > >
> > > I am planning to put a new Rtools online today that uses a different
> > > build of gcc 4.9.2.  I will be concentrating on getting it to work with
> > > all the external libraries before the 3.2.0 release next month.  I'm not
> > > planning to try to get it to work with R-patched, and I expect it won't:
> > >  I needed to make a number of patches to R-devel for compatibility.
> >
> > I also worked off R-devel (I said wrongly that it was R-patched in my
> > original post) and benefited from your compatibility changes.
> >
> > I look forward to the new Rtools and will test it by compiling some packages.
>
> It's now on the main site at CRAN, and should propagate to the mirrors
> reasonably quickly.  I'm hoping that tomorrow's R-devel build will use
> it, but there may be some last minute problems.

I think R-devel will have some trouble finding the compilers if they
aren't in the default install locations in c:\Rtools.  This should be
fixed in a few days, probably by requiring an environment variable to
specify where Rtools is installed.

In the meantime, if you get "gcc not found" errors, you can manually
edit the etc/*/Makeconf file in the R-devel binary install, and set
BINPREF to  <Rtools>/gcc492_32/bin/ for *=i386, and set BINPREF64 to
<Rtools>/gcc492_64/bin/ for *=x64.

Duncan Murdoch

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Hsiu-Khuern Tang-2
In reply to this post by Duncan Murdoch-2
Hi Duncan,

On Mon, Mar 9, 2015 at 10:40 AM, Duncan Murdoch
<[hidden email]> wrote:

> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
>>
>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]>
>> wrote:
>> > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>> >> Hi,
>> >>
>> >> [This is a follow-up to the "New version of Rtools for Windows" thread
>> >> in January, but I just subscribed and don't know how to reply to an
>> >> old thread -- my apologies.]
>> >
>> > I am planning to put a new Rtools online today that uses a different
>> > build of gcc 4.9.2.  I will be concentrating on getting it to work with
>> > all the external libraries before the 3.2.0 release next month.  I'm not
>> > planning to try to get it to work with R-patched, and I expect it won't:
>> >  I needed to make a number of patches to R-devel for compatibility.
>>
>> I also worked off R-devel (I said wrongly that it was R-patched in my
>> original post) and benefited from your compatibility changes.
>>
>> I look forward to the new Rtools and will test it by compiling some
>> packages.
>
>
> It's now on the main site at CRAN, and should propagate to the mirrors
> reasonably quickly.  I'm hoping that tomorrow's R-devel build will use it,
> but there may be some last minute problems.

Is the new Rtools at
http://cran.r-project.org/bin/windows/Rtools/Rtools33.exe?  I'm still
getting "Error 404 object not found".

Thanks,
- Hsiu-Khuern

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Avraham Adler
> On Mon, Mar 9, 2015 at 10:40 AM, Duncan Murdoch
> <[hidden email]> wrote:
> It's now on the main site at CRAN, and should propagate to the mirrors
> reasonably quickly.  I'm hoping that tomorrow's R-devel build will use it,
> but there may be some last minute problems.

Using Rtools 3.3, once it propagated through the cran servers, I have
successfully built a 64-bit version of R on Windows 7, up through make
rinstaller. This one includes using ICU_531, and also includes linking
to 64-bit OpenBLAS 2.13 (4 threads).

As with yesterday's build using 4.9.2.-seh (although that one left ICU
out) the only issue that seems to have failed in make check-all is the
internet connectivity, which is disabled by default. Loading R and
passing setinternet2() fixes that, and I plan on using the options
built into the installer I create to have that set at install (like
SDI). Is it at all possible to have that setting exposed in
Mkrules.dist so as to be set at compile?

I also built microbenchmark, which requires packages ‘colorspace’,
‘Rcpp’, ‘stringr’, ‘RColorBrewer’, ‘dichromat’, ‘munsell’, ‘labeling’,
‘plyr’, ‘digest’, ‘gtable’, ‘reshape2’, ‘scales’, ‘proto’, and
‘ggplot2’, and they all worked fine. For what it is worth, I forgot to
uncomment (unhash) Hsiu-Khuern's addition to the NM filter, yet Rcpp
built fine and compiled C++ code fine as well, although about 3%-5%
slower than what I recall from last night's seh version.

So, outside of this hiccup with somehow now needing internet2 (which
may have to do with microsoft Windows patches for all I know) which
cannot be set at default, it seems as if the toolchain is behaving
well! I have not tried building with curl, though; that looks a bit
more hairy, although it may address the internet2 issue, who knows.

For interest sakes, below is a comparison of speed across various
versions/compilers which may prove of interest. The takeaway for me is
that for matrix code a fast BLAS is significantly more important than
which version of GCC and exception handling is used. For non-BLAS
specific code, at least on my machine, the SJLJ performed about 1%–2%
*faster*. Go figure! Maybe someone will run Simon Urbanek's benchmark
against them.

Regardless, I'm much less apprehensive about 3.2's release in April.
Thank you, Duncan and all!


Avi



== Speed results compiled over a few months (except for the last two) ==

For the record, all code run on an Intel i7-2600K overclocked to
4.6Ghz, 16GB RAM, Windows 7 64bit Matrices A and B are 1000x1000 dense
matrices, of which A is positive semi-definite and B is not. I use
this to test BLAS builds. I hope that the fixed width works in plain
text model.

=== Non-BLAS dependent ===

#Test code
library(microbenchmark)
A <- as.matrix(read.csv(file="F:/R/A.csv", colClasses='numeric'))
B <- as.matrix(read.csv(file="F:/R/B.csv", colClasses='numeric'))
colnames(A) <- colnames(B) <- NULL
Z <- microbenchmark(A + 2, A - 2, A * 2, A / 2, A + B, A - B, A * B, A
/ B, A ^ 2, sqrt(A), control=list(order = 'block'), times = 1000L)


 R-devel_2015-03-08 compiled using
x86_64-4.9.2-release-win32-seh-rt_v3-rev1 (EOPTS = -O3 -march=native
-mfpmath=sse -msse2avx -mavx256-split-unaligned-load
-mavx256-split-unaligned-store -mvzeroupper -std=gnu++11 -pipe)
 OpenBLAS 2.13 - Multi-threaded (max 4 threads) - compiled under GCC
4.9.1 (MinGW-64)

 Unit: microseconds
    expr       min        lq      mean    median        uq      max neval
   A + 2   923.001  1844.215  2205.385  1858.957  1990.900 21714.18  1000
   A - 2  1742.652  1830.215  2196.901  1844.810  2507.798 21778.22  1000
   A * 2  1743.247  1843.023  2208.374  1860.298  2547.112 21776.43  1000
     A/2  2025.598  2111.375  2438.503  2122.097  2701.243 22034.06  1000
   A + B  2016.662  2124.182  2554.006  2143.690  2948.896 21964.07  1000
   A - B  2004.153  2103.930  2527.219  2128.203  2982.552 22295.27  1000
   A * B  2023.215  2119.715  2540.680  2141.010  3154.553 22074.27  1000
     A/B  3256.265  3354.700  3633.556  3368.252  3953.950 23189.67  1000
     A^2  1745.332  1835.279  2204.023  1850.469  2554.856 21869.66  1000
 sqrt(A) 49945.064 50066.434 50506.344 50187.356 50883.403 70006.25  1000


R-devel_2015-03-09 compiled using Rtools 3.3 (GCC 4.9.2, SJLJ, EOPTS =
-O3 -march=native -mfpmath=sse -msse2avx -mavx256-split-unaligned-load
-mavx256-split-unaligned-store -mvzeroupper -std=gnu++11 -pipe)
OpenBLAS 2.13 - Multi-threaded (max 4 threads) - compiled under GCC
4.9.1 (MinGW-64)

Unit: microseconds
    expr       min        lq      mean    median        uq      max neval
   A + 2   925.980  1777.350  2167.326  1791.795  2384.641 21660.28  1000
   A - 2  1673.256  1777.648  2188.756  1806.687  2670.715 21724.01  1000
   A * 2  1680.999  1786.434  2221.432  1835.130  2766.916 22254.16  1000
     A/2  1992.836  2085.165  2450.455  2108.694  2865.203 22803.08  1000
   A + B  1977.646  2089.632  2559.912  2121.204  3031.397 22884.99  1000
   A - B  1979.135  2081.591  2516.943  2101.398  3003.548 22377.77  1000
   A * B  1971.689  2073.699  2510.912  2092.462  2921.345 22308.37  1000
     A/B  3247.031  3345.169  3633.351  3361.402  3941.590 23231.97  1000
     A^2  1668.788  1771.244  2169.422  1788.220  2745.026 21786.86  1000
 sqrt(A) 48662.871 48805.537 49357.270 49003.003 49715.283 69269.10  1000


=== BLAS dependent code (statistics gathered over a few months ===

#Test code
library(microbenchmark)
library(Matrix)
A <- as.matrix(read.csv(file="F:/R/A.csv", colClasses='numeric'))
B <- as.matrix(read.csv(file="F:/R/B.csv", colClasses='numeric'))
colnames(A) <- colnames(B) <- NULL

Z <- microbenchmark(
    sort(A),
    t(A) %*% B,
    crossprod(A, B),
    solve(A),
    solve(A, diag(A)),
    chol(A),
    chol(B, pivot = TRUE),
    qr(A, LAPACK=TRUE),
    svd(A),
    eigen(A, symmetric = TRUE),
    eigen(A, symmetric = FALSE),
    eigen(B, symmetric = FALSE),
    lu(A),
    fft(A),
    times=100L, unit='ms', control = list(order = 'block'))


REFERENCE 3.1.1 compiled using Rtools 3.1 (GCC 4.6.3, default EOPTS flags)
reference BLAS

Unit: milliseconds
                        expr         min          lq        mean
median          uq         max neval
                     sort(A)   89.364120   90.760662   95.096270
91.561537   92.573725  154.081306   100
                  t(A) %*% B  463.145756  470.406496  487.680120
474.872066  490.043866  642.640917   100
             crossprod(A, B)  727.114903  729.128111  730.031458
729.785877  731.120320  733.078130   100
                    solve(A)  600.629979  604.814394  630.598703
608.606561  658.326032  662.879314   100
           solve(A, diag(A))  145.738089  146.774104  147.629655
147.959780  148.371535  148.883512   100
                     chol(A)  115.873110  116.019644  117.347118
116.212938  118.026150  172.853468   100
       chol(B, pivot = TRUE)    2.415134    2.548564    3.227905
2.559286    4.568473    4.689393   100
        qr(A, LAPACK = TRUE)  414.455301  416.033671  418.583569
416.972741  417.814271  473.541941   100
                      svd(A) 1952.765952 1957.070246 1974.547371
1959.374735 2010.263499 2017.405106   100
  eigen(A, symmetric = TRUE)  917.120317  920.482414  923.423802
921.784990  924.577926  980.692929   100
 eigen(A, symmetric = FALSE) 2981.049436 2985.640691 3007.526012
2991.149276 3014.926832 3130.924137   100
 eigen(B, symmetric = FALSE) 3964.874086 3974.978839 3999.080880
3991.973829 4019.799690 4078.083071   100
                       lu(A)  137.437464  138.229850  141.696849
138.906528  142.217546  198.202991   100
                      fft(A)  109.981065  110.321042  111.753592
110.640916  111.268152  116.670410   100


3.1.2 compiled using Rtools 3.2 (GCC 4.6.3, EOPTS = -march=native -O3
-std=gnu++0x -msse2avx -mavx256-split-unaligned-load
-mavx256-split-unaligned-store -mvzeroupper --param
l1-cache-line-size=64 --param l1-cache-size=64 --param
l2-cache-size=256)
OpenBLAS 2.13 - Multi-threaded (max 4 threads) - compiled under GCC
4.9.1 (MinGW-64)

Unit: milliseconds
                        expr         min          lq        mean
median          uq         max neval
                     sort(A)   88.771066   89.748265   94.542642
90.596947   91.482709  149.171214   100
                  t(A) %*% B   27.507195   33.359067   40.378088
37.689446   41.512909   96.868916   100
             crossprod(A, B)   17.783759   22.327538   26.787467
27.059399   31.918288   36.209055   100
                    solve(A)   45.964657   54.856090   80.761447
60.499775  109.150759  118.817308   100
           solve(A, diag(A))   24.704266   26.370058   26.805694
26.936840   27.400868   29.522052   100
                     chol(A)    6.762058    7.088337    8.725137
8.145653    8.973040   65.570275   100
       chol(B, pivot = TRUE)    2.558110    2.702412    3.481314
2.831076    4.789643    5.346446   100
        qr(A, LAPACK = TRUE)   78.757538   81.620631   85.132413
82.940043   85.099350  141.434937   100
                      svd(A)  361.539846  366.637747  386.533779
370.769323  421.736275  445.087770   100
  eigen(A, symmetric = TRUE)  174.249560  180.402841  186.649060
182.628715  188.931063  241.414148   100
 eigen(A, symmetric = FALSE)  734.881721  744.303748  772.203936
751.104077  795.883051  915.351575   100
 eigen(B, symmetric = FALSE) 2522.750166 2551.112148 2596.798329
2581.940655 2633.440287 2861.722717   100
                       lu(A)   20.277535   21.227185   25.068971
23.319926   25.130468   84.837552   100
                      fft(A)  109.757747  110.347313  112.123488
110.725415  114.057152  120.250492   100


R-devel_2015-03-09 compiled using Rtools 3.3 (GCC 4.9.2, SJLJ, EOPTS =
-O3 -march=native -mfpmath=sse -msse2avx -mavx256-split-unaligned-load
-mavx256-split-unaligned-store -mvzeroupper -std=gnu++11 -pipe)
OpenBLAS 2.13 - Multi-threaded (max 4 threads) - compiled under GCC
4.9.1 (MinGW-64)

Unit: milliseconds
                        expr         min          lq        mean
median          uq        max neval
                     sort(A)   88.025153   88.255828   92.701967
89.571826   90.320888  146.40380   100
                  t(A) %*% B   26.471552   30.866301   35.293662
34.069253   38.490212   85.57007   100
             crossprod(A, B)   17.606699   17.898879   23.999433
22.228699   28.620007   37.06744   100
                    solve(A)   43.410199   48.448279   54.914690
51.338798   55.865639  116.81746   100
           solve(A, diag(A))   24.655633   25.414227   27.692980
27.301179   28.757458   38.95692   100
                     chol(A)    6.620942    6.891379    8.010618
7.474695    8.233586   12.62357   100
       chol(B, pivot = TRUE)    2.456867    2.541751    3.737836
2.575556    2.722390   61.46246   100
        qr(A, LAPACK = TRUE)   78.153905   80.980389   83.663278
82.458112   84.998671  101.89696   100
                      svd(A)  353.204099  365.191932  390.446252
377.001957  417.792818  475.73975   100
  eigen(A, symmetric = TRUE)  173.627391  177.985954  186.068097
182.131711  187.866286  251.19902   100
 eigen(A, symmetric = FALSE)  771.643075  788.242038  813.902106
801.689427  839.380539  921.24119   100
 eigen(B, symmetric = FALSE) 2591.501370 2644.449833 2691.339277
2678.241053 2722.924657 2935.76884   100
                       lu(A)   19.969747   20.959164   24.298874
22.426017   24.017664   81.95253   100
                      fft(A)  106.862816  107.191480  108.985064
107.466682  110.465762  115.73511   100


 R-devel_2015-03-08 compiled using
x86_64-4.9.2-release-win32-seh-rt_v3-rev1 (EOPTS = -O3 -march=native
-mfpmath=sse -msse2avx -mavx256-split-unaligned-load
-mavx256-split-unaligned-store -mvzeroupper -std=gnu++11 -pipe)
 OpenBLAS 2.13 - Multi-threaded (max 4 threads) - compiled under GCC
4.9.1 (MinGW-64)

 Unit: milliseconds
                        expr         min          lq        mean
median          uq        max neval
                     sort(A)   88.372432   88.811892   93.321491
90.093638   90.754540  150.02760   100
                  t(A) %*% B   26.583837   30.443074   34.765044
33.903505   37.455374   82.54761   100
             crossprod(A, B)   17.715707   22.088566   26.875521
27.185023   31.154311   36.72850   100
                    solve(A)   44.112203   49.217298   55.707862
52.651668   57.331152  116.44069   100
           solve(A, diag(A))   24.891819   25.468731   27.590369
27.302520   29.217172   37.90168   100
                     chol(A)    6.658469    6.872168    7.893779
7.058167    8.968203   13.32230   100
       chol(B, pivot = TRUE)    2.451208    2.529540    3.742339
2.578981    2.646143   62.62224   100
        qr(A, LAPACK = TRUE)   78.839230   80.413602   82.989497
81.778148   84.447373   98.13199   100
                      svd(A)  352.931278  362.746235  387.952468
374.631166  415.481743  500.52405   100
  eigen(A, symmetric = TRUE)  172.696946  178.109557  187.816872
181.375053  190.414291  256.44276   100
 eigen(A, symmetric = FALSE)  778.904964  793.941318  820.598107
812.244809  841.944627  919.02527   100
 eigen(B, symmetric = FALSE) 2494.645617 2514.200623 2562.484197
2561.112354 2586.092481 2806.00525   100
                       lu(A)   19.762154   20.663114   24.555941
22.403382   24.369411   80.98218   100
                      fft(A)  106.374956  107.120148  108.625520
107.433176  108.786850  116.43563   100

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
In reply to this post by Hsiu-Khuern Tang-2
On 09/03/2015 11:02 PM, Hsiu-Khuern Tang wrote:

> Hi Duncan,
>
> On Mon, Mar 9, 2015 at 10:40 AM, Duncan Murdoch
> <[hidden email]> wrote:
>> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
>>>
>>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]>
>>> wrote:
>>>> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>>>>> Hi,
>>>>>
>>>>> [This is a follow-up to the "New version of Rtools for Windows" thread
>>>>> in January, but I just subscribed and don't know how to reply to an
>>>>> old thread -- my apologies.]
>>>>
>>>> I am planning to put a new Rtools online today that uses a different
>>>> build of gcc 4.9.2.  I will be concentrating on getting it to work with
>>>> all the external libraries before the 3.2.0 release next month.  I'm not
>>>> planning to try to get it to work with R-patched, and I expect it won't:
>>>>  I needed to make a number of patches to R-devel for compatibility.
>>>
>>> I also worked off R-devel (I said wrongly that it was R-patched in my
>>> original post) and benefited from your compatibility changes.
>>>
>>> I look forward to the new Rtools and will test it by compiling some
>>> packages.
>>
>>
>> It's now on the main site at CRAN, and should propagate to the mirrors
>> reasonably quickly.  I'm hoping that tomorrow's R-devel build will use it,
>> but there may be some last minute problems.
>
> Is the new Rtools at
> http://cran.r-project.org/bin/windows/Rtools/Rtools33.exe?  I'm still
> getting "Error 404 object not found".

There were some permission problems on the file for a while yesterday;
perhaps the index page got propagated but the actual file didn't.

Duncan Murdoch

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Dan Tenenbaum-2
In reply to this post by Duncan Murdoch-2


----- Original Message -----

> From: "Duncan Murdoch" <[hidden email]>
> To: "Hsiu-Khuern Tang" <[hidden email]>, [hidden email]
> Sent: Monday, March 9, 2015 10:40:02 AM
> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools (but not multilib)
>
> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> > On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
> > <[hidden email]> wrote:
> > > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> > >> Hi,
> > >>
> > >> [This is a follow-up to the "New version of Rtools for Windows"
> > >> thread
> > >> in January, but I just subscribed and don't know how to reply to
> > >> an
> > >> old thread -- my apologies.]
> > >
> > > I am planning to put a new Rtools online today that uses a
> > > different
> > > build of gcc 4.9.2.  I will be concentrating on getting it to
> > > work with
> > > all the external libraries before the 3.2.0 release next month.
> > >  I'm not
> > > planning to try to get it to work with R-patched, and I expect it
> > > won't:
> > >  I needed to make a number of patches to R-devel for
> > >  compatibility.
> >
> > I also worked off R-devel (I said wrongly that it was R-patched in
> > my
> > original post) and benefited from your compatibility changes.
> >
> > I look forward to the new Rtools and will test it by compiling some
> > packages.
>
> It's now on the main site at CRAN, and should propagate to the
> mirrors
> reasonably quickly.  I'm hoping that tomorrow's R-devel build will
> use
> it, but there may be some last minute problems.
>

Thanks to you and everyone who worked on this. Is there a way to tell which toolchain built a given R-devel binary?
If not, can you let us know when there is one on CRAN that was built with the new Rtools?

Thanks,
Dan


> Duncan Murdoch
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Hsiu-Khuern Tang-2
In reply to this post by Duncan Murdoch-2
On Tue, Mar 10, 2015 at 4:07 AM, Duncan Murdoch
<[hidden email]> wrote:

> On 09/03/2015 11:02 PM, Hsiu-Khuern Tang wrote:
>> Hi Duncan,
>>
>> On Mon, Mar 9, 2015 at 10:40 AM, Duncan Murdoch
>> <[hidden email]> wrote:
>>> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
>>>>
>>>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]>
>>>> wrote:
>>>>> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>>>>>> Hi,
>>>>>>
>>>>>> [This is a follow-up to the "New version of Rtools for Windows" thread
>>>>>> in January, but I just subscribed and don't know how to reply to an
>>>>>> old thread -- my apologies.]
>>>>>
>>>>> I am planning to put a new Rtools online today that uses a different
>>>>> build of gcc 4.9.2.  I will be concentrating on getting it to work with
>>>>> all the external libraries before the 3.2.0 release next month.  I'm not
>>>>> planning to try to get it to work with R-patched, and I expect it won't:
>>>>>  I needed to make a number of patches to R-devel for compatibility.
>>>>
>>>> I also worked off R-devel (I said wrongly that it was R-patched in my
>>>> original post) and benefited from your compatibility changes.
>>>>
>>>> I look forward to the new Rtools and will test it by compiling some
>>>> packages.
>>>
>>>
>>> It's now on the main site at CRAN, and should propagate to the mirrors
>>> reasonably quickly.  I'm hoping that tomorrow's R-devel build will use it,
>>> but there may be some last minute problems.
>>
>> Is the new Rtools at
>> http://cran.r-project.org/bin/windows/Rtools/Rtools33.exe?  I'm still
>> getting "Error 404 object not found".
>
> There were some permission problems on the file for a while yesterday;
> perhaps the index page got propagated but the actual file didn't.
>
> Duncan Murdoch
>

Got it now, thanks!  Are you planning to publish the build scripts for
the new Rtools as well?

I did the following limited test on the new Rtools:

R CMD INSTALL --no-multiarch Rcpp_0.11.5.tar.gz

I did this under various settings:

- the default settings
- with -std=c++11 added to CXXFLAGS in my .R\Makevars file.

This was done under the newly released R-3.1.3, using the 64-bit R binary.

Here are my findings:

- With the default settings, the command succeeded

- With -std=c++11, there were two problems:
  + api.cpp failed to compile because it could not find execinfo.h
    * I worked around this by using CXXFLAGS=-DWIN32 -std=c++11
    * CXXFLAGS=-std=gnu++11 also works around this
    * Maybe Rcpp needs to guard against this?
  + the package could not be loaded because some of the object files
contain symbols named .refptr.* and .weak.*, which should be excluded
from the exports list
    * To work around this, put this line in .R\Makevars: NM_FILTER = |
sed -e '/\.refptr\./d; /\.weak\./d'


- Hsiu-Khuern

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
In reply to this post by Dan Tenenbaum-2
On 10/03/2015 12:54 PM, Dan Tenenbaum wrote:

>
> ----- Original Message -----
> > From: "Duncan Murdoch" <[hidden email]>
> > To: "Hsiu-Khuern Tang" <[hidden email]>, [hidden email]
> > Sent: Monday, March 9, 2015 10:40:02 AM
> > Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools (but not multilib)
> >
> > On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> > > On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
> > > <[hidden email]> wrote:
> > > > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> > > >> Hi,
> > > >>
> > > >> [This is a follow-up to the "New version of Rtools for Windows"
> > > >> thread
> > > >> in January, but I just subscribed and don't know how to reply to
> > > >> an
> > > >> old thread -- my apologies.]
> > > >
> > > > I am planning to put a new Rtools online today that uses a
> > > > different
> > > > build of gcc 4.9.2.  I will be concentrating on getting it to
> > > > work with
> > > > all the external libraries before the 3.2.0 release next month.
> > > >  I'm not
> > > > planning to try to get it to work with R-patched, and I expect it
> > > > won't:
> > > >  I needed to make a number of patches to R-devel for
> > > >  compatibility.
> > >
> > > I also worked off R-devel (I said wrongly that it was R-patched in
> > > my
> > > original post) and benefited from your compatibility changes.
> > >
> > > I look forward to the new Rtools and will test it by compiling some
> > > packages.
> >
> > It's now on the main site at CRAN, and should propagate to the
> > mirrors
> > reasonably quickly.  I'm hoping that tomorrow's R-devel build will
> > use
> > it, but there may be some last minute problems.
> >
>
> Thanks to you and everyone who worked on this. Is there a way to tell which toolchain built a given R-devel binary?
> If not, can you let us know when there is one on CRAN that was built with the new Rtools?

If you look in etc/*/Makeconf, you'll see something like this:

BINPREF ?= $(RTOOLS)gcc492_64/bin/

hopefully from tomorrow onwards.  If today's build was with the new
toolchain, you should see a hardcoded path to where I have Rtools
installed on the build machine, which isn't so helpful.  The previous
toolchain left BINPREF blank.

If you want to use your own toolchain, just edit those files.  If you
want to install the standard Rtools somewhere else, set an environment
variable like

RTOOLS = C:/Rtools/

(where the terminal / is required.)

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
In reply to this post by Hsiu-Khuern Tang-2
On 10/03/2015 2:26 PM, Hsiu-Khuern Tang wrote:

> On Tue, Mar 10, 2015 at 4:07 AM, Duncan Murdoch
> <[hidden email]> wrote:
> > On 09/03/2015 11:02 PM, Hsiu-Khuern Tang wrote:
> >> Hi Duncan,
> >>
> >> On Mon, Mar 9, 2015 at 10:40 AM, Duncan Murdoch
> >> <[hidden email]> wrote:
> >>> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> >>>>
> >>>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch <[hidden email]>
> >>>> wrote:
> >>>>> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> [This is a follow-up to the "New version of Rtools for Windows" thread
> >>>>>> in January, but I just subscribed and don't know how to reply to an
> >>>>>> old thread -- my apologies.]
> >>>>>
> >>>>> I am planning to put a new Rtools online today that uses a different
> >>>>> build of gcc 4.9.2.  I will be concentrating on getting it to work with
> >>>>> all the external libraries before the 3.2.0 release next month.  I'm not
> >>>>> planning to try to get it to work with R-patched, and I expect it won't:
> >>>>>  I needed to make a number of patches to R-devel for compatibility.
> >>>>
> >>>> I also worked off R-devel (I said wrongly that it was R-patched in my
> >>>> original post) and benefited from your compatibility changes.
> >>>>
> >>>> I look forward to the new Rtools and will test it by compiling some
> >>>> packages.
> >>>
> >>>
> >>> It's now on the main site at CRAN, and should propagate to the mirrors
> >>> reasonably quickly.  I'm hoping that tomorrow's R-devel build will use it,
> >>> but there may be some last minute problems.
> >>
> >> Is the new Rtools at
> >> http://cran.r-project.org/bin/windows/Rtools/Rtools33.exe?  I'm still
> >> getting "Error 404 object not found".
> >
> > There were some permission problems on the file for a while yesterday;
> > perhaps the index page got propagated but the actual file didn't.
> >
> > Duncan Murdoch
> >
>
> Got it now, thanks!  Are you planning to publish the build scripts for
> the new Rtools as well?

Yes, they are on pre-CRAN now (that's where I put things, CRAN picks
them up from there, the mirrors pick them up from CRAN).  Not sure when
they'll propagate, but the URL is

bin/windows/Rtools/scripts

I'm going to be uploading a new Rtools33.exe in a few minutes.  It puts
back gdb, which had been included in the 4.6.3 build but not this one;
it is now in Rtools/bin as gdb.exe for 32 bits and gdb64.exe for 64
bits.  It also cleans up the PATH editing; the new scheme doesn't need
gcc to be on the PATH.  And the index page points to the scripts.

Duncan Murdoch

>
> I did the following limited test on the new Rtools:
>
> R CMD INSTALL --no-multiarch Rcpp_0.11.5.tar.gz
>
> I did this under various settings:
>
> - the default settings
> - with -std=c++11 added to CXXFLAGS in my .R\Makevars file.
>
> This was done under the newly released R-3.1.3, using the 64-bit R binary.
>
> Here are my findings:
>
> - With the default settings, the command succeeded
>
> - With -std=c++11, there were two problems:
>    + api.cpp failed to compile because it could not find execinfo.h
>      * I worked around this by using CXXFLAGS=-DWIN32 -std=c++11
>      * CXXFLAGS=-std=gnu++11 also works around this
>      * Maybe Rcpp needs to guard against this?
>    + the package could not be loaded because some of the object files
> contain symbols named .refptr.* and .weak.*, which should be excluded
> from the exports list
>      * To work around this, put this line in .R\Makevars: NM_FILTER = |
> sed -e '/\.refptr\./d; /\.weak\./d'
>
>
> - Hsiu-Khuern
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Dan Tenenbaum-2
In reply to this post by Duncan Murdoch-2


----- Original Message -----

> From: "Duncan Murdoch" <[hidden email]>
> To: "Dan Tenenbaum" <[hidden email]>
> Cc: "Hsiu-Khuern Tang" <[hidden email]>, [hidden email]
> Sent: Tuesday, March 10, 2015 11:37:12 AM
> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools (but not multilib)
>
> On 10/03/2015 12:54 PM, Dan Tenenbaum wrote:
> >
> > ----- Original Message -----
> > > From: "Duncan Murdoch" <[hidden email]>
> > > To: "Hsiu-Khuern Tang" <[hidden email]>, [hidden email]
> > > Sent: Monday, March 9, 2015 10:40:02 AM
> > > Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools
> > > (but not multilib)
> > >
> > > On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
> > > > On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
> > > > <[hidden email]> wrote:
> > > > > On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
> > > > >> Hi,
> > > > >>
> > > > >> [This is a follow-up to the "New version of Rtools for
> > > > >> Windows"
> > > > >> thread
> > > > >> in January, but I just subscribed and don't know how to
> > > > >> reply to
> > > > >> an
> > > > >> old thread -- my apologies.]
> > > > >
> > > > > I am planning to put a new Rtools online today that uses a
> > > > > different
> > > > > build of gcc 4.9.2.  I will be concentrating on getting it to
> > > > > work with
> > > > > all the external libraries before the 3.2.0 release next
> > > > > month.
> > > > >  I'm not
> > > > > planning to try to get it to work with R-patched, and I
> > > > > expect it
> > > > > won't:
> > > > >  I needed to make a number of patches to R-devel for
> > > > >  compatibility.
> > > >
> > > > I also worked off R-devel (I said wrongly that it was R-patched
> > > > in
> > > > my
> > > > original post) and benefited from your compatibility changes.
> > > >
> > > > I look forward to the new Rtools and will test it by compiling
> > > > some
> > > > packages.
> > >
> > > It's now on the main site at CRAN, and should propagate to the
> > > mirrors
> > > reasonably quickly.  I'm hoping that tomorrow's R-devel build
> > > will
> > > use
> > > it, but there may be some last minute problems.
> > >
> >
> > Thanks to you and everyone who worked on this. Is there a way to
> > tell which toolchain built a given R-devel binary?
> > If not, can you let us know when there is one on CRAN that was
> > built with the new Rtools?
>
> If you look in etc/*/Makeconf, you'll see something like this:
>
> BINPREF ?= $(RTOOLS)gcc492_64/bin/
>
> hopefully from tomorrow onwards.  If today's build was with the new
> toolchain, you should see a hardcoded path to where I have Rtools
> installed on the build machine, which isn't so helpful.  The previous
> toolchain left BINPREF blank.
>
> If you want to use your own toolchain, just edit those files.  If you
> want to install the standard Rtools somewhere else, set an
> environment
> variable like
>
> RTOOLS = C:/Rtools/
>
> (where the terminal / is required.)
>

Thanks, that's very helpful. I also notice with the latest R-devel binary (67969, which is built with the new Rtools according to what you say above) that I need to do setInternet2(TRUE) before I can download from any URLs; I see some mention of that earlier in this thread, is this intended? If so is there a way to make this the default?

Thanks,
Dan


>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Duncan Murdoch-2
On 10/03/2015 2:56 PM, Dan Tenenbaum wrote:

>
>
> ----- Original Message -----
>> From: "Duncan Murdoch" <[hidden email]>
>> To: "Dan Tenenbaum" <[hidden email]>
>> Cc: "Hsiu-Khuern Tang" <[hidden email]>, [hidden email]
>> Sent: Tuesday, March 10, 2015 11:37:12 AM
>> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools (but not multilib)
>>
>> On 10/03/2015 12:54 PM, Dan Tenenbaum wrote:
>>>
>>> ----- Original Message -----
>>>> From: "Duncan Murdoch" <[hidden email]>
>>>> To: "Hsiu-Khuern Tang" <[hidden email]>, [hidden email]
>>>> Sent: Monday, March 9, 2015 10:40:02 AM
>>>> Subject: Re: [Rd] Notes on building a gcc toolchain for Rtools
>>>> (but not multilib)
>>>>
>>>> On 09/03/2015 11:07 AM, Hsiu-Khuern Tang wrote:
>>>>> On Mon, Mar 9, 2015 at 3:50 AM, Duncan Murdoch
>>>>> <[hidden email]> wrote:
>>>>>> On 08/03/2015 10:02 PM, Hsiu-Khuern Tang wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> [This is a follow-up to the "New version of Rtools for
>>>>>>> Windows"
>>>>>>> thread
>>>>>>> in January, but I just subscribed and don't know how to
>>>>>>> reply to
>>>>>>> an
>>>>>>> old thread -- my apologies.]
>>>>>>
>>>>>> I am planning to put a new Rtools online today that uses a
>>>>>> different
>>>>>> build of gcc 4.9.2.  I will be concentrating on getting it to
>>>>>> work with
>>>>>> all the external libraries before the 3.2.0 release next
>>>>>> month.
>>>>>>  I'm not
>>>>>> planning to try to get it to work with R-patched, and I
>>>>>> expect it
>>>>>> won't:
>>>>>>  I needed to make a number of patches to R-devel for
>>>>>>  compatibility.
>>>>>
>>>>> I also worked off R-devel (I said wrongly that it was R-patched
>>>>> in
>>>>> my
>>>>> original post) and benefited from your compatibility changes.
>>>>>
>>>>> I look forward to the new Rtools and will test it by compiling
>>>>> some
>>>>> packages.
>>>>
>>>> It's now on the main site at CRAN, and should propagate to the
>>>> mirrors
>>>> reasonably quickly.  I'm hoping that tomorrow's R-devel build
>>>> will
>>>> use
>>>> it, but there may be some last minute problems.
>>>>
>>>
>>> Thanks to you and everyone who worked on this. Is there a way to
>>> tell which toolchain built a given R-devel binary?
>>> If not, can you let us know when there is one on CRAN that was
>>> built with the new Rtools?
>>
>> If you look in etc/*/Makeconf, you'll see something like this:
>>
>> BINPREF ?= $(RTOOLS)gcc492_64/bin/
>>
>> hopefully from tomorrow onwards.  If today's build was with the new
>> toolchain, you should see a hardcoded path to where I have Rtools
>> installed on the build machine, which isn't so helpful.  The previous
>> toolchain left BINPREF blank.
>>
>> If you want to use your own toolchain, just edit those files.  If you
>> want to install the standard Rtools somewhere else, set an
>> environment
>> variable like
>>
>> RTOOLS = C:/Rtools/
>>
>> (where the terminal / is required.)
>>
>
> Thanks, that's very helpful. I also notice with the latest R-devel binary (67969, which is built with the new Rtools according to what you say above) that I need to do setInternet2(TRUE) before I can download from any URLs; I see some mention of that earlier in this thread, is this intended? If so is there a way to make this the default?

That's a bug.  I haven't tracked down what's going wrong with our
regular code.  If I can't find that and fix it soon, I'll make Internet2
the default.  For now, you can do it yourself using the instructions on
?setInternet2.

Duncan Murdoch

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Notes on building a gcc toolchain for Rtools (but not multilib)

Avraham Adler
On Tue, Mar 10, 2015 at 3:17 PM, Duncan Murdoch
<[hidden email]> wrote:
>
> That's a bug.  I haven't tracked down what's going wrong with our
> regular code.  If I can't find that and fix it soon, I'll make Internet2
> the default.  For now, you can do it yourself using the instructions on
> ?setInternet2.
>
> Duncan Murdoch

I successfully rebuilt R-devel_2015-03-09 with the most recent version
of Rtools tonight, building both ICU_531 and this time libcurl (7.39)
as well (and OpenBLAS, of course). The internet bug is still there,
but the rest of make-check all passed with flying colors, as did
building 'microbenchmark' from source (with all the other needed
packages, including Rcpp and Hsiu-Kheurn's change to NM was *not*
used). My non-BLAS test tonight ran faster than last night; maybe 1000
iterations aren't enough or I had something else eating up clock
cycles last night. Either way, outside the internet bug, it's looking
good for Windows 64bit (Win7 at least).

Thanks,

Avi

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
12