gfortran 9 quantreg bug

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

gfortran 9 quantreg bug

Roger Koenker-3
I’d like to solicit some advice on a debugging problem I have in the quantreg package.
Kurt and Brian have reported to me that on Debian machines with gfortran 9

library(quantreg)
f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
plot(f)

fails because summary() produces bogus estimates of the coefficient bounds.
This example has been around in my R package from the earliest days of R, and
before that in various incarnations of S.  The culprit is apparently rqbr.f which is
even more ancient, but must have something that gfortran 9 doesn’t approve of.

I note that in R-devel there have been some other issues with gfortran 9, but these seem
unrelated to my problem.  Not having access to a machine with an R/gfortran9
configuration, I can’t  apply my rudimentary debugging methods.  I’ve considered
trying to build gfortran on my mac air and then building R from source, but before
going down this road, I wondered whether others had other suggestions, or
advice about  my proposed route.  As far as I can see there are not yet
binaries for gfortran 9 for osx.

Thanks,
Roger

Roger Koenker
[hidden email]<mailto:[hidden email]>
Department of Economics, UCL
London  WC1H 0AX.



        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Berend Hasselman
Roger,

I have run

        gfortran -c -fsyntax-only -fimplicit-none -Wall -pedantic rqbr.f

in the src folder of quantreg.

There are many warnings about defined but not used labels.
Also two errors such as "Symbol ‘in’ at (1) has no IMPLICIT type".
And warnings such as: Warning: "Possible change of value in conversion from REAL(8) to INTEGER(4)  at ..."

No offense intended but this fortran code is awful. I wouldn't want to debug this before an extensive cleanup by
getting rid of as many numerical labels as possible, indenting and doing something about the warnings "Possible change of value ...".

This is going to be very difficult.

Berend Hasselman

> On 4 Aug 2019, at 08:48, Koenker, Roger W <[hidden email]> wrote:
>
> I’d like to solicit some advice on a debugging problem I have in the quantreg package.
> Kurt and Brian have reported to me that on Debian machines with gfortran 9
>
> library(quantreg)
> f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
> plot(f)
>
> fails because summary() produces bogus estimates of the coefficient bounds.
> This example has been around in my R package from the earliest days of R, and
> before that in various incarnations of S.  The culprit is apparently rqbr.f which is
> even more ancient, but must have something that gfortran 9 doesn’t approve of.
>
> I note that in R-devel there have been some other issues with gfortran 9, but these seem
> unrelated to my problem.  Not having access to a machine with an R/gfortran9
> configuration, I can’t  apply my rudimentary debugging methods.  I’ve considered
> trying to build gfortran on my mac air and then building R from source, but before
> going down this road, I wondered whether others had other suggestions, or
> advice about  my proposed route.  As far as I can see there are not yet
> binaries for gfortran 9 for osx.
>
> Thanks,
> Roger
>
> Roger Koenker
> [hidden email]<mailto:[hidden email]>
> Department of Economics, UCL
> London  WC1H 0AX.
>
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Dirk Eddelbuettel
In reply to this post by Roger Koenker-3

Roger,

On 4 August 2019 at 06:48, Koenker, Roger W wrote:
| I’d like to solicit some advice on a debugging problem I have in the quantreg package.
| Kurt and Brian have reported to me that on Debian machines with gfortran 9
|
| library(quantreg)
| f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
| plot(f)
|
| fails because summary() produces bogus estimates of the coefficient bounds.
| This example has been around in my R package from the earliest days of R, and
| before that in various incarnations of S.  The culprit is apparently rqbr.f which is
| even more ancient, but must have something that gfortran 9 doesn’t approve of.
|
| I note that in R-devel there have been some other issues with gfortran 9, but these seem
| unrelated to my problem.  Not having access to a machine with an R/gfortran9
| configuration, I can’t  apply my rudimentary debugging methods.  I’ve considered
| trying to build gfortran on my mac air and then building R from source, but before
| going down this road, I wondered whether others had other suggestions, or
| advice about  my proposed route.  As far as I can see there are not yet
| binaries for gfortran 9 for osx.

Maybe installing and running Docker on your mac is an alternative?

Minimally viable example using

  a) docker (on Linux, but it is portable) and
 
  b) the current official 'r-base' container (an alias to our Rocker r-base container)

r-base is begged to Debian testing, and also allows you to get Debian
unstable.  Below I fire up the container, tell it to use bash (not R) and update

  edd@rob:~/git$ docker run --rm -ti r-base bash
  root@1307193fadf4:/#
  root@1307193fadf4:/# apt-get update
  Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [149 kB]
  Get:2 http://cdn-fastly.deb.debian.org/debian testing InRelease [117 kB]
  Get:3 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages [8,385 kB]
  Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64 Packages [7,918 kB]
  Fetched 16.6 MB in 4s (4,649 kB/s)                          
  Reading package lists... Done
  root@1307193fadf4:/# apt-cache policy gcc-9
  gcc-9:
    Installed: (none)
    Candidate: 9.1.0-10
    Version table:
       9.1.0-10 990
          990 http://deb.debian.org/debian testing/main amd64 Packages
          500 http://http.debian.net/debian sid/main amd64 Packages
  root@1307193fadf4:/# apt-cache policy gfortran-9
  gfortran-9:
    Installed: (none)
    Candidate: 9.1.0-10
    Version table:
       9.1.0-10 990
          990 http://deb.debian.org/debian testing/main amd64 Packages
          500 http://http.debian.net/debian sid/main amd64 Packages
  root@1307193fadf4:/#

At this point it just a matter of actually installing gcc-9 and gfortran-9
(via apt-get install ...), and setting CC, FC, F77 and whichever other
environment variables the R build reflect to build quantreg.

That said, this will be Debian's standard gfortran-9.  What is at times a
little frustrating is that some of the builds used by some of the CRAN tests
use local modifications which make their behaviour a little harder to
reproduce.  I have an open issue with my (also old and stable) digest package
which goes belly-up on a clang-on-Fedora build and nowhere else -- and I have
been unable to reproduce this too.

For such cases, having Docker container would be one possible way of
giving access to the test environment to make it accessible to more users.

Best,  Dirk


|
| Thanks,
| Roger
|
| Roger Koenker
| [hidden email]<mailto:[hidden email]>
| Department of Economics, UCL
| London  WC1H 0AX.
|
|
|
| [[alternative HTML version deleted]]
|
| ______________________________________________
| [hidden email] mailing list
| https://stat.ethz.ch/mailman/listinfo/r-devel

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Balasubramanian Narasimhan
In reply to this post by Berend Hasselman

On 8/4/19 7:26 AM, Berend Hasselman wrote:

> Roger,
>
> I have run
>
> gfortran -c -fsyntax-only -fimplicit-none -Wall -pedantic rqbr.f
>
> in the src folder of quantreg.
>
> There are many warnings about defined but not used labels.
> Also two errors such as "Symbol ‘in’ at (1) has no IMPLICIT type".
> And warnings such as: Warning: "Possible change of value in conversion from REAL(8) to INTEGER(4)  at ..."
>
> No offense intended but this fortran code is awful. I wouldn't want to debug this before an extensive cleanup by
> getting rid of as many numerical labels as possible, indenting and doing something about the warnings "Possible change of value ...".

The unused labels at least can be removed automatically at least for
fixed form along the lines shown in steps 8 and 9 of

https://bnaras.github.io/SUtools/articles/SUtools.html

which pertain to lines 261--281 of

https://github.com/bnaras/SUtools/blob/master/R/process.R

In fact, here it is, excerpted.

library(stringr)
code_lines  <- readLines(con = "rqbr.f")
cat("Running gfortran to detect warning lines on unused labels\n")
system2(command = "gfortran",
         args = c("-Wunused", "-c", "rqbr.f", "-o", "temp.o"),
         stderr = "gfortran.out")
cat("Scanning gfortran output for warnings on unusued labels\n")
warnings <- readLines("gfortran.out")
line_numbers <- grep('rqbr.f', warnings)
label_warning_line_numbers <- grep(pattern = "^Warning: Label [0-9]+ at", warnings)
just_warnings <- sum(grepl('Warning:', warnings))

nW <- length(label_warning_line_numbers)
for (i in seq_len(nW)) {
     offending_line <- as.integer(stringr::str_extract(warnings[line_numbers[i]], pattern = "([0-9]+)"))
     code_line <- code_lines[offending_line]
     offending_label <- stringr::str_extract(warnings[label_warning_line_numbers[i]],
                                             pattern = "([0-9]+)")
     code_lines[offending_line] <- sub(pattern = offending_label,
                                       replacement = str_pad("", width = nchar(offending_label)),
                                       x = code_lines[offending_line])
}
writeLines(code_lines, con = "rqbr-new.f")

-Naras

> This is going to be very difficult.
>
> Berend Hasselman
>
>> On 4 Aug 2019, at 08:48, Koenker, Roger W <[hidden email]> wrote:
>>
>> I’d like to solicit some advice on a debugging problem I have in the quantreg package.
>> Kurt and Brian have reported to me that on Debian machines with gfortran 9
>>
>> library(quantreg)
>> f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
>> plot(f)
>>
>> fails because summary() produces bogus estimates of the coefficient bounds.
>> This example has been around in my R package from the earliest days of R, and
>> before that in various incarnations of S.  The culprit is apparently rqbr.f which is
>> even more ancient, but must have something that gfortran 9 doesn’t approve of.
>>
>> I note that in R-devel there have been some other issues with gfortran 9, but these seem
>> unrelated to my problem.  Not having access to a machine with an R/gfortran9
>> configuration, I can’t  apply my rudimentary debugging methods.  I’ve considered
>> trying to build gfortran on my mac air and then building R from source, but before
>> going down this road, I wondered whether others had other suggestions, or
>> advice about  my proposed route.  As far as I can see there are not yet
>> binaries for gfortran 9 for osx.
>>
>> Thanks,
>> Roger
>>
>> Roger Koenker
>> [hidden email]<mailto:[hidden email]>
>> Department of Economics, UCL
>> London  WC1H 0AX.
>>
>>
>>
>> [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Roger Koenker-3
In reply to this post by Berend Hasselman
Thanks Berend,

Yes,  I know about these warnings, they are mostly a consequence of the automated
translation from the ancient Bell Labs dialect of fortran called ratfor.  It is easy to add
type declarations for “in”  and the others, but it seems unlikely that this is going to fix
anything.  The extra labels are all attributable to the ratfor translation.  I agree that
the code is ugly — the ratfor is somewhat better, but not much.  I fact the algorithm
is rather simple, but I’m reluctant to write it again from scratch, since there are few
fiddly details and I would worry somewhat about reproducibility.

Roger

Roger Koenker
[hidden email]<mailto:[hidden email]>
Department of Economics, UCL
London  WC1H 0AX.


On Aug 4, 2019, at 3:26 PM, Berend Hasselman <[hidden email]<mailto:[hidden email]>> wrote:

Roger,

I have run

gfortran -c -fsyntax-only -fimplicit-none -Wall -pedantic rqbr.f

in the src folder of quantreg.

There are many warnings about defined but not used labels.
Also two errors such as "Symbol ‘in’ at (1) has no IMPLICIT type".
And warnings such as: Warning: "Possible change of value in conversion from REAL(8) to INTEGER(4)  at ..."

No offense intended but this fortran code is awful. I wouldn't want to debug this before an extensive cleanup by
getting rid of as many numerical labels as possible, indenting and doing something about the warnings "Possible change of value ...".

This is going to be very difficult.

Berend Hasselman

On 4 Aug 2019, at 08:48, Koenker, Roger W <[hidden email]<mailto:[hidden email]>> wrote:

I’d like to solicit some advice on a debugging problem I have in the quantreg package.
Kurt and Brian have reported to me that on Debian machines with gfortran 9

library(quantreg)
f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
plot(f)

fails because summary() produces bogus estimates of the coefficient bounds.
This example has been around in my R package from the earliest days of R, and
before that in various incarnations of S.  The culprit is apparently rqbr.f which is
even more ancient, but must have something that gfortran 9 doesn’t approve of.

I note that in R-devel there have been some other issues with gfortran 9, but these seem
unrelated to my problem.  Not having access to a machine with an R/gfortran9
configuration, I can’t  apply my rudimentary debugging methods.  I’ve considered
trying to build gfortran on my mac air and then building R from source, but before
going down this road, I wondered whether others had other suggestions, or
advice about  my proposed route.  As far as I can see there are not yet
binaries for gfortran 9 for osx.

Thanks,
Roger

Roger Koenker
[hidden email]<mailto:[hidden email]><mailto:[hidden email]>
Department of Economics, UCL
London  WC1H 0AX.



[[alternative HTML version deleted]]

______________________________________________
[hidden email]<mailto:[hidden email]> mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Roger Koenker-3
In reply to this post by Dirk Eddelbuettel
With extensive help from Dirk Eddelbuettel I have installed
docker on my mac mini from

    https://hub.docker.com/editions/community/docker-ce-desktop-mac

which installs from a dmg in quite standard fashion.  This has allowed
me to simulate running R in a Debian environment with gfortran-9 and
begin the process of debugging my ancient rqbr.f code.

Some further details:

0.  After some initial testing, e.g.

    docker --version
    docker run hello-world      

1.  Download r-base and test os

    docker pull r-base                       $ downloads r-base for us
    docker run --rm -ti r-base R --version   # to check we have the R we want
    docker run --rm -ti r-base bash          # now in shell, Ctrl-d to exit

2.  Setup working directory -- tell Docker to run from the current directory
    and access

    cd projects/rq
    docker run --rm -ti -v ${PWD}:/work -w /work r-base bash

    This put the contents of projects/rq into the /work directory.

    root@90521904fa86:/work# apt-get update
    Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [149 kB]
    Get:2 http://cdn-fastly.deb.debian.org/debian testing InRelease [117 kB]
    Get:3 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages [8,385 kB]
    Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64 Packages [7,916 kB]
    Fetched 16.6 MB in 4s (4,411 kB/s)                          
    Reading package lists... Done
 
3.  Get gcc-9 and gfortran-9

    root@90521904fa86:/work# apt-get install gcc-9 gfortran-9
    Reading package lists... Done
    Building dependency tree      
    Reading state information... Done
    The following additional packages will be installed:
    cpp-9 gcc-9-base libasan5 libatomic1 libcc1-0 libgcc-9-dev libgcc1 libgfortran-9-dev
    libgfortran5 libgomp1 libitm1 liblsan0 libquadmath0 libstdc++6 libtsan0 libubsan1
    Suggested packages:
    gcc-9-locales gcc-9-multilib gcc-9-doc libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg
    libasan5-dbg liblsan0-dbg libtsan0-dbg libubsan1-dbg libquadmath0-dbg gfortran-9-multilib
    gfortran-9-doc libgfortran5-dbg libcoarrays-dev
    The following NEW packages will be installed:
    cpp-9 gcc-9 gfortran-9 libgcc-9-dev libgfortran-9-dev
    The following packages will be upgraded:
    gcc-9-base libasan5 libatomic1 libcc1-0 libgcc1 libgfortran5 libgomp1 libitm1 liblsan0
    libquadmath0 libstdc++6 libtsan0 libubsan1
    13 upgraded, 5 newly installed, 0 to remove and 71 not upgraded.
    Need to get 35.6 MB of archives.
    After this operation, 107 MB of additional disk space will be used.
    Do you want to continue? [Y/n] Y
    Get:1 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libasan5 amd64 9.1.0-10 [390 kB]
    Get:2 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libubsan1 amd64 9.1.0-10 [128 kB]
    Get:3 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libtsan0 amd64 9.1.0-10 [295 kB]
    Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64 gcc-9-base amd64 9.1.0-10 [190 kB]
    Get:5 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libstdc++6 amd64 9.1.0-10 [500 kB]
    Get:6 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libquadmath0 amd64 9.1.0-10 [145 kB]
    Get:7 http://cdn-fastly.deb.debian.org/debian testing/main amd64 liblsan0 amd64 9.1.0-10 [137 kB]
    Get:8 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libitm1 amd64 9.1.0-10 [27.6 kB]
    Get:9 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgomp1 amd64 9.1.0-10 [88.1 kB]
    Get:10 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgfortran5 amd64 9.1.0-10 [633 kB]
    Get:11 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libcc1-0 amd64 9.1.0-10 [47.7 kB]
    Get:12 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libatomic1 amd64 9.1.0-10 [9,012 B]
    Get:13 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgcc1 amd64 1:9.1.0-10 [40.5 kB]
    Get:14 http://cdn-fastly.deb.debian.org/debian testing/main amd64 cpp-9 amd64 9.1.0-10 [9,667 kB]
    Get:15 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgcc-9-dev amd64 9.1.0-10 [2,346 kB]
    Get:16 http://cdn-fastly.deb.debian.org/debian testing/main amd64 gcc-9 amd64 9.1.0-10 [9,945 kB]
    Get:17 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgfortran-9-dev amd64 9.1.0-10 [676 kB]
    Get:18 http://cdn-fastly.deb.debian.org/debian testing/main amd64 gfortran-9 amd64 9.1.0-10 [10.4 MB]
    Fetched 35.6 MB in 6s (6,216 kB/s)      
    debconf: delaying package configuration, since apt-utils is not installed
    (Reading database ... 17787 files and directories currently installed.)
    Preparing to unpack .../libasan5_9.1.0-10_amd64.deb ...
    Unpacking libasan5:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../libubsan1_9.1.0-10_amd64.deb ...
    Unpacking libubsan1:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../libtsan0_9.1.0-10_amd64.deb ...
    Unpacking libtsan0:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../gcc-9-base_9.1.0-10_amd64.deb ...
    Unpacking gcc-9-base:amd64 (9.1.0-10) over (9.1.0-8) ...
    Setting up gcc-9-base:amd64 (9.1.0-10) ...
    (Reading database ... 17787 files and directories currently installed.)
    Preparing to unpack .../libstdc++6_9.1.0-10_amd64.deb ...
    Unpacking libstdc++6:amd64 (9.1.0-10) over (9.1.0-8) ...
    Setting up libstdc++6:amd64 (9.1.0-10) ...
    (Reading database ... 17787 files and directories currently installed.)
    Preparing to unpack .../0-libquadmath0_9.1.0-10_amd64.deb ...
    Unpacking libquadmath0:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../1-liblsan0_9.1.0-10_amd64.deb ...
    Unpacking liblsan0:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../2-libitm1_9.1.0-10_amd64.deb ...
    Unpacking libitm1:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../3-libgomp1_9.1.0-10_amd64.deb ...
    Unpacking libgomp1:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../4-libgfortran5_9.1.0-10_amd64.deb ...
    Unpacking libgfortran5:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../5-libcc1-0_9.1.0-10_amd64.deb ...
    Unpacking libcc1-0:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../6-libatomic1_9.1.0-10_amd64.deb ...
    Unpacking libatomic1:amd64 (9.1.0-10) over (9.1.0-8) ...
    Preparing to unpack .../7-libgcc1_1%3a9.1.0-10_amd64.deb ...
    Unpacking libgcc1:amd64 (1:9.1.0-10) over (1:9.1.0-8) ...
    Setting up libgcc1:amd64 (1:9.1.0-10) ...
    Selecting previously unselected package cpp-9.
    (Reading database ... 17787 files and directories currently installed.)
    Preparing to unpack .../cpp-9_9.1.0-10_amd64.deb ...
    Unpacking cpp-9 (9.1.0-10) ...
    Selecting previously unselected package libgcc-9-dev:amd64.
    Preparing to unpack .../libgcc-9-dev_9.1.0-10_amd64.deb ...
    Unpacking libgcc-9-dev:amd64 (9.1.0-10) ...
    Selecting previously unselected package gcc-9.
    Preparing to unpack .../gcc-9_9.1.0-10_amd64.deb ...
    Unpacking gcc-9 (9.1.0-10) ...
    Selecting previously unselected package libgfortran-9-dev:amd64.
    Preparing to unpack .../libgfortran-9-dev_9.1.0-10_amd64.deb ...
    Unpacking libgfortran-9-dev:amd64 (9.1.0-10) ...
    Selecting previously unselected package gfortran-9.
    Preparing to unpack .../gfortran-9_9.1.0-10_amd64.deb ...
    Unpacking gfortran-9 (9.1.0-10) ...
    Setting up libgomp1:amd64 (9.1.0-10) ...
    Setting up libasan5:amd64 (9.1.0-10) ...
    Setting up libquadmath0:amd64 (9.1.0-10) ...
    Setting up libatomic1:amd64 (9.1.0-10) ...
    Setting up libgfortran5:amd64 (9.1.0-10) ...
    Setting up libubsan1:amd64 (9.1.0-10) ...
    Setting up cpp-9 (9.1.0-10) ...
    Setting up libcc1-0:amd64 (9.1.0-10) ...
    Setting up liblsan0:amd64 (9.1.0-10) ...
    Setting up libitm1:amd64 (9.1.0-10) ...
    Setting up libtsan0:amd64 (9.1.0-10) ...
    Setting up libgcc-9-dev:amd64 (9.1.0-10) ...
    Setting up gcc-9 (9.1.0-10) ...
    Setting up libgfortran-9-dev:amd64 (9.1.0-10) ...
    Setting up gfortran-9 (9.1.0-10) ...
    Processing triggers for libc-bin (2.28-10) ...
    root@90521904fa86:/work# pwd
   
4.  At this point I removed some dependencies from the package
    quantreg that I knew were not relevant to the debugging problem at
    hand. This included an attempt to set compiler flags in
    quantreg/src/Makevars, but this didn't work.

5.  Set compiler flags as follows:

    root@90521904fa86:/work# mkdir ~/.R; vi ~/.R/Makevars

        CC=gcc-9
        FC=gfortran-9
        F77=gfortran-9

    Alternatively, one can find the settings of CC, FC, CXX, ... in /etc/R/Makeconf
    and alter them there.
   
6.  At this point

    R CMD INSTALL quantreg_5.43.tar.gz

    did use the gfortran-9  compiler and this version did reproduce the
    error initially reported by Kurt and Brian.  

7.  Now it is (just!) a matter of finding the bug.
     
Roger Koenker
[hidden email]
Department of Economics, UCL
London  WC1H 0AX.


> On Aug 4, 2019, at 3:41 PM, Dirk Eddelbuettel <[hidden email]> wrote:
>
>
> Roger,
>
> On 4 August 2019 at 06:48, Koenker, Roger W wrote:
> | I’d like to solicit some advice on a debugging problem I have in the quantreg package.
> | Kurt and Brian have reported to me that on Debian machines with gfortran 9
> |
> | library(quantreg)
> | f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
> | plot(f)
> |
> | fails because summary() produces bogus estimates of the coefficient bounds.
> | This example has been around in my R package from the earliest days of R, and
> | before that in various incarnations of S.  The culprit is apparently rqbr.f which is
> | even more ancient, but must have something that gfortran 9 doesn’t approve of.
> |
> | I note that in R-devel there have been some other issues with gfortran 9, but these seem
> | unrelated to my problem.  Not having access to a machine with an R/gfortran9
> | configuration, I can’t  apply my rudimentary debugging methods.  I’ve considered
> | trying to build gfortran on my mac air and then building R from source, but before
> | going down this road, I wondered whether others had other suggestions, or
> | advice about  my proposed route.  As far as I can see there are not yet
> | binaries for gfortran 9 for osx.
>
> Maybe installing and running Docker on your mac is an alternative?
>
> Minimally viable example using
>
>  a) docker (on Linux, but it is portable) and
>
>  b) the current official 'r-base' container (an alias to our Rocker r-base container)
>
> r-base is begged to Debian testing, and also allows you to get Debian
> unstable.  Below I fire up the container, tell it to use bash (not R) and update
>
>  edd@rob:~/git$ docker run --rm -ti r-base bash
>  root@1307193fadf4:/#
>  root@1307193fadf4:/# apt-get update
>  Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [149 kB]
>  Get:2 http://cdn-fastly.deb.debian.org/debian testing InRelease [117 kB]
>  Get:3 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages [8,385 kB]
>  Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64 Packages [7,918 kB]
>  Fetched 16.6 MB in 4s (4,649 kB/s)                          
>  Reading package lists... Done
>  root@1307193fadf4:/# apt-cache policy gcc-9
>  gcc-9:
>    Installed: (none)
>    Candidate: 9.1.0-10
>    Version table:
>       9.1.0-10 990
>          990 http://deb.debian.org/debian testing/main amd64 Packages
>          500 http://http.debian.net/debian sid/main amd64 Packages
>  root@1307193fadf4:/# apt-cache policy gfortran-9
>  gfortran-9:
>    Installed: (none)
>    Candidate: 9.1.0-10
>    Version table:
>       9.1.0-10 990
>          990 http://deb.debian.org/debian testing/main amd64 Packages
>          500 http://http.debian.net/debian sid/main amd64 Packages
>  root@1307193fadf4:/#
>
> At this point it just a matter of actually installing gcc-9 and gfortran-9
> (via apt-get install ...), and setting CC, FC, F77 and whichever other
> environment variables the R build reflect to build quantreg.
>
> That said, this will be Debian's standard gfortran-9.  What is at times a
> little frustrating is that some of the builds used by some of the CRAN tests
> use local modifications which make their behaviour a little harder to
> reproduce.  I have an open issue with my (also old and stable) digest package
> which goes belly-up on a clang-on-Fedora build and nowhere else -- and I have
> been unable to reproduce this too.
>
> For such cases, having Docker container would be one possible way of
> giving access to the test environment to make it accessible to more users.
>
> Best,  Dirk
>
>
> |
> | Thanks,
> | Roger
> |
> | Roger Koenker
> | [hidden email]<mailto:[hidden email]>
> | Department of Economics, UCL
> | London  WC1H 0AX.
> |
> |
> |
> | [[alternative HTML version deleted]]
> |
> | ______________________________________________
> | [hidden email] mailing list
> | https://stat.ethz.ch/mailman/listinfo/r-devel
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Kasper Daniel Hansen-2
In reply to this post by Dirk Eddelbuettel
Dirk,

Thanks for the blog post on this, and the pointers in this email.

I have a question: it seems to me that you end up using a different
compiler for the package (quantreg) than was used to build R itself. As I
understand ABI changes, this is considered unsupported (ok, that depends on
what version of gcc/gfortran was used to build R, but there has been a lot
of ABI changes in GCC). Is that correct? I understand that this shortcut
makes it much easier to use different compilers, and might work for Roger's
usecase, but I was wondering about this issue of using a different compiler
for packages. Is this something I should worry about?

Best,
Kasper

On Sun, Aug 4, 2019 at 10:41 AM Dirk Eddelbuettel <[hidden email]> wrote:

>
> Roger,
>
> On 4 August 2019 at 06:48, Koenker, Roger W wrote:
> | I’d like to solicit some advice on a debugging problem I have in the
> quantreg package.
> | Kurt and Brian have reported to me that on Debian machines with gfortran
> 9
> |
> | library(quantreg)
> | f = summary(rq(foodexp ~ income, data = engel, tau = 1:4/5))
> | plot(f)
> |
> | fails because summary() produces bogus estimates of the coefficient
> bounds.
> | This example has been around in my R package from the earliest days of
> R, and
> | before that in various incarnations of S.  The culprit is apparently
> rqbr.f which is
> | even more ancient, but must have something that gfortran 9 doesn’t
> approve of.
> |
> | I note that in R-devel there have been some other issues with gfortran
> 9, but these seem
> | unrelated to my problem.  Not having access to a machine with an
> R/gfortran9
> | configuration, I can’t  apply my rudimentary debugging methods.  I’ve
> considered
> | trying to build gfortran on my mac air and then building R from source,
> but before
> | going down this road, I wondered whether others had other suggestions, or
> | advice about  my proposed route.  As far as I can see there are not yet
> | binaries for gfortran 9 for osx.
>
> Maybe installing and running Docker on your mac is an alternative?
>
> Minimally viable example using
>
>   a) docker (on Linux, but it is portable) and
>
>   b) the current official 'r-base' container (an alias to our Rocker
> r-base container)
>
> r-base is begged to Debian testing, and also allows you to get Debian
> unstable.  Below I fire up the container, tell it to use bash (not R) and
> update
>
>   edd@rob:~/git$ docker run --rm -ti r-base bash
>   root@1307193fadf4:/#
>   root@1307193fadf4:/# apt-get update
>   Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [149 kB]
>   Get:2 http://cdn-fastly.deb.debian.org/debian testing InRelease [117 kB]
>   Get:3 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages
> [8,385 kB]
>   Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64
> Packages [7,918 kB]
>   Fetched 16.6 MB in 4s (4,649 kB/s)
>   Reading package lists... Done
>   root@1307193fadf4:/# apt-cache policy gcc-9
>   gcc-9:
>     Installed: (none)
>     Candidate: 9.1.0-10
>     Version table:
>        9.1.0-10 990
>           990 http://deb.debian.org/debian testing/main amd64 Packages
>           500 http://http.debian.net/debian sid/main amd64 Packages
>   root@1307193fadf4:/# apt-cache policy gfortran-9
>   gfortran-9:
>     Installed: (none)
>     Candidate: 9.1.0-10
>     Version table:
>        9.1.0-10 990
>           990 http://deb.debian.org/debian testing/main amd64 Packages
>           500 http://http.debian.net/debian sid/main amd64 Packages
>   root@1307193fadf4:/#
>
> At this point it just a matter of actually installing gcc-9 and gfortran-9
> (via apt-get install ...), and setting CC, FC, F77 and whichever other
> environment variables the R build reflect to build quantreg.
>
> That said, this will be Debian's standard gfortran-9.  What is at times a
> little frustrating is that some of the builds used by some of the CRAN
> tests
> use local modifications which make their behaviour a little harder to
> reproduce.  I have an open issue with my (also old and stable) digest
> package
> which goes belly-up on a clang-on-Fedora build and nowhere else -- and I
> have
> been unable to reproduce this too.
>
> For such cases, having Docker container would be one possible way of
> giving access to the test environment to make it accessible to more users.
>
> Best,  Dirk
>
>
> |
> | Thanks,
> | Roger
> |
> | Roger Koenker
> | [hidden email]<mailto:[hidden email]>
> | Department of Economics, UCL
> | London  WC1H 0AX.
> |
> |
> |
> |       [[alternative HTML version deleted]]
> |
> | ______________________________________________
> | [hidden email] mailing list
> | https://stat.ethz.ch/mailman/listinfo/r-devel
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


--
Best,
Kasper

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Dirk Eddelbuettel

Hi Kasper,

On 6 August 2019 at 10:33, Kasper Daniel Hansen wrote:
| Thanks for the blog post on this, and the pointers in this email.

My pleasure!  There wasn't much in there that was "new" but it often helps to
just tie it together with a valid and real example (as provided by Roger).
 
| I have a question: it seems to me that you end up using a different
| compiler for the package (quantreg) than was used to build R itself. As I
| understand ABI changes, this is considered unsupported (ok, that depends on

Are you thinking that because the 'number' increased to 9, the ABI must have
changed in some (presumably incompatible) ways?  Luckily that is not
generally the case or we'd be in much dire straits.

Every couple of years there is such a change but it is rare. I can't even
recall when g++ last forced us.  May have been the 3.3 to 4.0 change with 3.4
then providing compatibility with what came after. As for Fortran, can't
recall such a change.

| what version of gcc/gfortran was used to build R, but there has been a lot
| of ABI changes in GCC). Is that correct? I understand that this shortcut
| makes it much easier to use different compilers, and might work for Roger's
| usecase, but I was wondering about this issue of using a different compiler
| for packages. Is this something I should worry about?

No.

But don't take my word for it, but trust 'R CMD check'. If it tests, trust it.

Hth, Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: gfortran 9 quantreg bug

Dirk Eddelbuettel

On 6 August 2019 at 10:51, Dirk Eddelbuettel wrote:
| then providing compatibility with what came after. As for Fortran, can't
| recall such a change.

Come to think about it we had it in Debian once or twice in the 20+ years I
contributed but I can't recall anymore when it was either.

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel