Test suite failures in R-devel_2017-02-25_r72256

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Test suite failures in R-devel_2017-02-25_r72256

Peter Simons-2
Hi,

I tried compiling the latest pre-release for R 3.3.3 for the NixOS Linux
distribution [1], but the build fails during the "make check" phase
because of the following 2 issues:

1) The "tools" test in "tests/Examples" requires network access, which
   it doesn't have in our build environment. Therefore, it fails as
   follows according to "tools-Ex.Rout.fail":

   | [...]
   | > set.seed(11)
   | > ## End(Don't show)
   | > pdb <- CRAN_package_db()
   | Warning in url(sprintf("%s/%s", cran, path), open = "rb") :
   |   URL 'http://CRAN.R-project.org/web/packages/packages.rds': status was 'Couldn't resolve host name'
   | Error in url(sprintf("%s/%s", cran, path), open = "rb") :
   |   cannot open the connection to 'http://CRAN.R-project.org/web/packages/packages.rds'
   | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object -> gzcon -> url
   | Execution halted

   I'm wondering whether it would be possible to extend the test suite
   with a configure-time flag that disable tests which depend on network
   access? My experience is that most modern Linux distributions run
   their builds in a restricted environment and therefore will run into
   trouble if the suite assumes that it can access the Internet.

2) When R is compiled with the --without-recommended-packages flag
   (which is our preferred configuration), the "base" test in
   "tests/Examples" fails, apparently because it depends on MASS. I'm
   citing from "base-Ex.Rout.fail":

   | >  ## The string "foo" and the symbol 'foo' can be used interchangably here:
   | >  stopifnot( identical(isNamespaceLoaded(  "foo"   ), FALSE),
   | +             identical(isNamespaceLoaded(quote(foo)), FALSE),
   | +             identical(isNamespaceLoaded(quote(stats)), statL))
   | >
   | > hasM <- isNamespaceLoaded("MASS") # (to restore if needed)
   | > Mns <- asNamespace("MASS") # loads it if not already
   | Error in loadNamespace(name) : there is no package called 'MASS'
   | Calls: asNamespace ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
   | Execution halted

I hope this helps!

Best regards,
Peter



[1] http://nixos.org/

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Test suite failures in R-devel_2017-02-25_r72256

Martin Maechler
>>>>> Peter Simons <[hidden email]>
>>>>>     on Mon, 27 Feb 2017 10:30:39 +0100 writes:

    > Hi, I tried compiling the latest pre-release for R 3.3.3
    > for the NixOS Linux distribution [1], but the build fails
    > during the "make check" phase because of the following 2
    > issues:

    > 1) The "tools" test in "tests/Examples" requires network
    > access, which it doesn't have in our build
    > environment.

One may argue that the 'make check' (or even 'make check-all')
tests could / should be allowed more resources than the pure
build environment.

    > Therefore, it fails as follows according to
    > "tools-Ex.Rout.fail":

    > | [...]
    > | > set.seed(11)
    > | > ## End(Don't show)
    > | > pdb <- CRAN_package_db()
    > | Warning in url(sprintf("%s/%s", cran, path), open = "rb") :
    > |   URL 'http://CRAN.R-project.org/web/packages/packages.rds': status was 'Couldn't resolve host name'
    > | Error in url(sprintf("%s/%s", cran, path), open = "rb") :
    > |   cannot open the connection to 'http://CRAN.R-project.org/web/packages/packages.rds'
    > | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object -> gzcon -> url
    > | Execution halted

    > I'm wondering whether it would be possible to extend the test suite
    > with a configure-time flag that disable tests which depend on network
    > access? My experience is that most modern Linux distributions run
    > their builds in a restricted environment and therefore will run into
    > trouble if the suite assumes that it can access the Internet.

[see above] Is it necessary to also run the 'make check' part in
that restricted environment?  Or could that ('checking") not get
more priviledges?

Note that you can only run  "make check" if you don't install
recommended packages, whereas more thorough testing would
include
        make check-devel
or even
        make check-all

but these do have quite a bit more requirements including
recommended packages being present.


    > 2) When R is compiled with the --without-recommended-packages flag
    > (which is our preferred configuration), the "base" test in
    > "tests/Examples" fails, apparently because it depends on MASS. I'm
    > citing from "base-Ex.Rout.fail":

    > | >  ## The string "foo" and the symbol 'foo' can be used interchangably here:
    > | >  stopifnot( identical(isNamespaceLoaded(  "foo"   ), FALSE),
    > | +             identical(isNamespaceLoaded(quote(foo)), FALSE),
    > | +             identical(isNamespaceLoaded(quote(stats)), statL))
    > | >
    > | > hasM <- isNamespaceLoaded("MASS") # (to restore if needed)
    > | > Mns <- asNamespace("MASS") # loads it if not already
    > | Error in loadNamespace(name) : there is no package called 'MASS'
    > | Calls: asNamespace ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
    > | Execution halted

Yes, that example should not have assumed a recommended package
to be available unconditionally.

I've changed it, thank you!


    > I hope this helps!

    > Best regards,
    > Peter



    > [1] http://nixos.org/

    > ______________________________________________
    > [hidden email] mailing list
    > https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Test suite failures in R-devel_2017-02-25_r72256

Martin Maechler
>>>>> Martin Maechler <[hidden email]>
>>>>>     on Mon, 27 Feb 2017 16:08:40 +0100 writes:

>>>>> Peter Simons <[hidden email]>
>>>>>     on Mon, 27 Feb 2017 10:30:39 +0100 writes:

    >> Hi, I tried compiling the latest pre-release for R 3.3.3
    >> for the NixOS Linux distribution [1], but the build fails
    >> during the "make check" phase because of the following 2
    >> issues:

    >> 1) The "tools" test in "tests/Examples" requires network
    >> access, which it doesn't have in our build
    >> environment.

    > One may argue that the 'make check' (or even 'make check-all')
    > tests could / should be allowed more resources than the pure
    > build environment.

    >> Therefore, it fails as follows according to
    >> "tools-Ex.Rout.fail":

    >> | [...]
    >> | > set.seed(11)
    >> | > ## End(Don't show)
    >> | > pdb <- CRAN_package_db()
    >> | Warning in url(sprintf("%s/%s", cran, path), open = "rb") :
    >> |   URL 'http://CRAN.R-project.org/web/packages/packages.rds': status was 'Couldn't resolve host name'
    >> | Error in url(sprintf("%s/%s", cran, path), open = "rb") :
    >> |   cannot open the connection to 'http://CRAN.R-project.org/web/packages/packages.rds'
    >> | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object -> gzcon -> url
    >> | Execution halted

    >> I'm wondering whether it would be possible to extend the test suite
    >> with a configure-time flag that disable tests which depend on network
    >> access? My experience is that most modern Linux distributions run
    >> their builds in a restricted environment and therefore will run into
    >> trouble if the suite assumes that it can access the Internet.

    > [see above] Is it necessary to also run the 'make check' part in
    > that restricted environment?  Or could that ('checking") not get
    > more priviledges?

    > Note that you can only run  "make check" if you don't install
    > recommended packages, whereas more thorough testing would
    > include
    >      make check-devel
    > or even
    >      make check-all
    >
    > but these do have quite a bit more requirements including
    > recommended packages being present.

I have to correct myself:  The above paragraph may be misleading:

Much, if not all of
      make check-devel
and   make check-all
have worked well since R version 3.1.0  which had in its NEWS an entry

     • More of 'make check' will work if recommended packages are not
       installed: but recommended packages remain needed for thorough
       checking of an R build.

Further, the 'R-admin' manual (on 'make check-all' etc) contains

 |  Note that these checks are only run completely
 |  if the recommended packages are installed.

so their presence is not required  but much recommended for
thorough testing.

Martin

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Test suite failures in R-devel_2017-02-25_r72256

Prof Brian Ripley
In reply to this post by Peter Simons-2
On 27/02/2017 09:30, Peter Simons wrote:
> Hi,
>
> I tried compiling the latest pre-release for R 3.3.3 for the NixOS Linux
> distribution [1], but the build fails during the "make check" phase
> because of the following 2 issues:

Hmm, R-devel (your subject line) is not a pre-release of R 3.3.3: it is
'R Under development' for what is planned as R 3.4.0.  Pre-release
tarballs for 3.3.3 are things like R-rc_2017-02-26_r72260.tar.gz (and
were previously labelled R-beta).

Your first point only occurs in R-devel, and was something already
reported and under investigation.  Note that test does not actually
depend on network access: it depends on having an accessible CRAN
mirror.  The latter could be local (and is on the CRAN check farm, for
example).  So we have been working on a more sophisticated condition to
run that test.

There are lots of possible check environments: it is very time-consuming
to check them all and reasonable coverage is only attempted once a
pre-release reaches 'alpha' status -- 3.4.0 has not done so but is
planned to be in late March.

I have just re-checked on Linux, and R 3.3.3 RC passed its checks
without Internet access (which is checked for where needed).

I also re-checked 3.3.3 RC without recommended packages and got the
expected messages about incomplete testing but no failures except in
'make check-recommended' (expected!).

Our posting guide does ask you to include the output of sessionInfo():
that would have avoided the version confusion (and might even have
alerted you to it).


> 1) The "tools" test in "tests/Examples" requires network access, which
>    it doesn't have in our build environment. Therefore, it fails as
>    follows according to "tools-Ex.Rout.fail":
>
>    | [...]
>    | > set.seed(11)
>    | > ## End(Don't show)
>    | > pdb <- CRAN_package_db()
>    | Warning in url(sprintf("%s/%s", cran, path), open = "rb") :
>    |   URL 'http://CRAN.R-project.org/web/packages/packages.rds': status was 'Couldn't resolve host name'
>    | Error in url(sprintf("%s/%s", cran, path), open = "rb") :
>    |   cannot open the connection to 'http://CRAN.R-project.org/web/packages/packages.rds'
>    | Calls: CRAN_package_db -> as.data.frame -> read_CRAN_object -> gzcon -> url
>    | Execution halted
>
>    I'm wondering whether it would be possible to extend the test suite
>    with a configure-time flag that disable tests which depend on network
>    access? My experience is that most modern Linux distributions run
>    their builds in a restricted environment and therefore will run into
>    trouble if the suite assumes that it can access the Internet.
>
> 2) When R is compiled with the --without-recommended-packages flag
>    (which is our preferred configuration), the "base" test in
>    "tests/Examples" fails, apparently because it depends on MASS. I'm
>    citing from "base-Ex.Rout.fail":
>
>    | >  ## The string "foo" and the symbol 'foo' can be used interchangably here:
>    | >  stopifnot( identical(isNamespaceLoaded(  "foo"   ), FALSE),
>    | +             identical(isNamespaceLoaded(quote(foo)), FALSE),
>    | +             identical(isNamespaceLoaded(quote(stats)), statL))
>    | >
>    | > hasM <- isNamespaceLoaded("MASS") # (to restore if needed)
>    | > Mns <- asNamespace("MASS") # loads it if not already
>    | Error in loadNamespace(name) : there is no package called 'MASS'
>    | Calls: asNamespace ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
>    | Execution halted
>
> I hope this helps!
>
> Best regards,
> Peter
>
>
>
> [1] http://nixos.org/
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


--
Brian D. Ripley,                  [hidden email]
Emeritus Professor of Applied Statistics, University of Oxford

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Test suite failures in R-rc_2017-02-28_r72286.tar.gz

Peter Simons-2
Hi Brian,

 > R-devel (your subject line) is not a pre-release of R 3.3.3: it is 'R
 > Under development' for what is planned as R 3.4.0. Pre-release
 > tarballs for 3.3.3 are things like R-rc_2017-02-26_r72260.tar.gz (and
 > were previously labelled R-beta).

I did not realize this distinction exists. I am sorry about the
misleading report. I followed the link for "3.3.3 release candidates"
and didn't expect that directory to contain tarballs for 3.4.0, too. In
hindsight, that should have been obvious from the different file names,
I suppose. Thank you for the clarification.

I re-ran the build and test suite with the correct tarball,
R-rc_2017-02-28_r72286.tar.gz, and the result looks different indeed.
With that version, I get a test suite error in reg-tests-1c from
"tests". I'm citing the relevant bit from "reg-tests-1c.Rout.fail":

 | > [...]
 | > ## format.POSIXlt() of Jan.1 if  1941 or '42 is involved:
 | > tJan1 <- function(n1, n2)
 | +     strptime(paste0(n1:n2,"/01/01"), "%Y/%m/%d", tz="CET")
 | > wDSTJan1 <- function(n1, n2)
 | +     which("CEST" == sub(".* ", '', format(tJan1(n1,n2), usetz=TRUE)))
 | > (w8 <- wDSTJan1(1801, 2300))
 | integer(0)
 | > (w9 <- wDSTJan1(1901, 2300))
 | integer(0)
 | > stopifnot(identical(w8, 141:142),# exactly 1941:1942 had CEST on Jan.1
 | +           identical(w9,  41: 42))
 | Error: identical(w8, 141:142) is not TRUE
 | Execution halted

This happens in the --without-recommended-packages style build. Not sure
whether this is relevant.

The sessionInfo() output looks as follows:

 | R version 3.3.3 RC (2017-02-28 r72286)
 | Platform: x86_64-pc-linux-gnu (64-bit)
 |
 | locale:
 | [1] C
 |
 | attached base packages:
 | [1] stats     graphics  grDevices utils     datasets  base    

Best regards,
Peter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Test suite failures in R-devel_2017-02-25_r72256

Peter Simons-2
In reply to this post by Martin Maechler
Hi Martin,

 > Is it necessary to also run the 'make check' part in that restricted
 > environment? Or could that ('checking") not get more priviledges?

in NixOS, there is no way to run any kind of build process in an
environment that has network access. On one hand, that is a security
consideration, but the even more compelling reason for that choice is
that NixOS goes to great lengths to guarantee deterministic builds, and
a build process that connects to servers on the Internet and uses data
they provides is as non-deterministic as it can be, unfortunately.

What we can do for the 3.4.0 release is to extend our build environment
for R to ensure that all required resources are downloaded and made
available locally before the build starts.

 > Note that you can only run  "make check" if you don't install
 > recommended packages, whereas more thorough testing would
 > include
 >      make check-devel
 > or even
 >      make check-all
 >
 > but these do have quite a bit more requirements including recommended
 > packages being present.

Thank you for pointing that out. We do offer our users the choice of
whether they want the recommended packages included or not (the default
choice being to not include them), so we should in fact parameterize our
build to run the extended test suite if the recommended packages are
built.

Another option would be to always build the recommended packages, but to
not install them if the user doesn't want them included. I'm not sure,
though, whether the R build system supports such a type of build easily?

Best regards,
Peter

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel