Byte-compilation failure on different architectures / low-memory systems

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Byte-compilation failure on different architectures / low-memory systems

Dirk Eddelbuettel

As you may know, I look after the R package for Debian. My fellow Debianers
make me follow a specific protocol -- a so-called "transition" in which all
dependent packages on an identified potential breakge are rebuilt under the
new (potentially breaking) change. We started adding an r-api-3.4 tag last
(major) release. Now it is r-api-3.5 and the transition got started with the
green light from the release team last Friday.

Now one build failure occurred for one of the (old, effectively litte
maintained) RMetrics packages: fBasics. It blows up at the byte-compilation
step on at least four (older, smaller) architectures: mips, mipsel, arm64
(ubuntu), ppc64el.  More at https://bugs.debian.org/900756 where we also
worked out the "solution" of suppressing byte-compilation at installation.

Luke, Tomas, ...: Would it help you to get access to such hardware?

We may get you some sort of "guest pass" access to porter machines. Else I
could try but I have my hands plenty full with the transition (having built
[and finally transferred to our Debian Gitlab instance] 20+ package during
day two of R/Finance...).

Cheers, Dirk

PS I CC'ed the bug report, if any follow-up keep the CC it gets logged there.

PPS Happy to discuss / help off-list too, of course.

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Byte-compilation failure on different architectures / low-memory systems

Hugh Parsonage
I believe a reproducible example is to simply have a very large object
defined literally (i.e. through structure() etc) in ./R/ . For
example, fBasics has a couple of files test-jbLM.R and test-jbTable.R
which are about 500 KB. For a given amount of RAM, I believe any
sufficiently large file will also break a package if byte compilation
is not turned off, regardless of architecture. A similar issue
occurred with package ddalpha, though the maintainer has since fixed
that I believe.

On 5 June 2018 at 01:14, Dirk Eddelbuettel <[hidden email]> wrote:

>
> As you may know, I look after the R package for Debian. My fellow Debianers
> make me follow a specific protocol -- a so-called "transition" in which all
> dependent packages on an identified potential breakge are rebuilt under the
> new (potentially breaking) change. We started adding an r-api-3.4 tag last
> (major) release. Now it is r-api-3.5 and the transition got started with the
> green light from the release team last Friday.
>
> Now one build failure occurred for one of the (old, effectively litte
> maintained) RMetrics packages: fBasics. It blows up at the byte-compilation
> step on at least four (older, smaller) architectures: mips, mipsel, arm64
> (ubuntu), ppc64el.  More at https://bugs.debian.org/900756 where we also
> worked out the "solution" of suppressing byte-compilation at installation.
>
> Luke, Tomas, ...: Would it help you to get access to such hardware?
>
> We may get you some sort of "guest pass" access to porter machines. Else I
> could try but I have my hands plenty full with the transition (having built
> [and finally transferred to our Debian Gitlab instance] 20+ package during
> day two of R/Finance...).
>
> Cheers, Dirk
>
> PS I CC'ed the bug report, if any follow-up keep the CC it gets logged there.
>
> PPS Happy to discuss / help off-list too, of course.
>
> --
> http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Byte-compilation failure on different architectures / low-memory systems

Tomas Kalibera
In reply to this post by Dirk Eddelbuettel
Hi Dirk,

thanks for the report. Access to the test system is not necessary, the
memory requirements of the byte-code compiler are usually
platform-independent and specifically with this package I can reproduce
they are very high. We'll have a look what we can do, certainly there
should at least be a way to recover and use the uncompiled version when
memory allocation fails, this is already done by the JIT but not when
compiling during installation. Before R or the package is patched, the
only workaround for memory constrained systems is probably to disable
byte-compilation of this package, as I read you are doing already.

Best
Tomas

On 06/04/2018 05:14 PM, Dirk Eddelbuettel wrote:

> As you may know, I look after the R package for Debian. My fellow Debianers
> make me follow a specific protocol -- a so-called "transition" in which all
> dependent packages on an identified potential breakge are rebuilt under the
> new (potentially breaking) change. We started adding an r-api-3.4 tag last
> (major) release. Now it is r-api-3.5 and the transition got started with the
> green light from the release team last Friday.
>
> Now one build failure occurred for one of the (old, effectively litte
> maintained) RMetrics packages: fBasics. It blows up at the byte-compilation
> step on at least four (older, smaller) architectures: mips, mipsel, arm64
> (ubuntu), ppc64el.  More at https://bugs.debian.org/900756 where we also
> worked out the "solution" of suppressing byte-compilation at installation.
>
> Luke, Tomas, ...: Would it help you to get access to such hardware?
>
> We may get you some sort of "guest pass" access to porter machines. Else I
> could try but I have my hands plenty full with the transition (having built
> [and finally transferred to our Debian Gitlab instance] 20+ package during
> day two of R/Finance...).
>
> Cheers, Dirk
>
> PS I CC'ed the bug report, if any follow-up keep the CC it gets logged there.
>
> PPS Happy to discuss / help off-list too, of course.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Byte-compilation failure on different architectures / low-memory systems

Dirk Eddelbuettel

On 4 June 2018 at 20:06, Tomas Kalibera wrote:
| thanks for the report. Access to the test system is not necessary, the
| memory requirements of the byte-code compiler are usually
| platform-independent and specifically with this package I can reproduce
| they are very high. We'll have a look what we can do, certainly there
| should at least be a way to recover and use the uncompiled version when
| memory allocation fails, this is already done by the JIT but not when
| compiling during installation. Before R or the package is patched, the
| only workaround for memory constrained systems is probably to disable
| byte-compilation of this package, as I read you are doing already.

Yes. And as a shortcut, we just turned it off unconditionally, ie on all
build architectures.  Worked fine as per

   https://buildd.debian.org/status/package.php?p=fbasics

it has been built everywhere where we have R 3.5.0 (some 20 or so platforms).

The fix you suggest sounds ideal: if possible recover, and maybe WARN.

Dirk

--
http://dirk.eddelbuettel.com | @eddelbuettel | [hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: Byte-compilation failure on different architectures / low-memory systems

Tomas Kalibera
It turned out slightly more complicated, the fallback to AST is was
already in place, but memory available to malloc got exhausted via heap
expansion. Messages starting "Error: compilation failed" mean there was
an error while compiling, including out of memory, and that the AST will
be used - so these are worth looking at but not fatal. The error message
"cannot allocate buffer" was the real problem.

We have addressed that with Luke in R-devel and R-patched (more general
fix might come later to R-devel only). Now there is a GC on the fallback
path which should release some memory from R heap to be usable by malloc
again. Also, the compiler needs a bit less memory when analyzing similar
code patterns as those causing trouble in fBasics (solves the fBasics
problems with "ulimit -v 1000000")

Best
Tomas

On 06/05/2018 08:59 PM, Dirk Eddelbuettel wrote:

> On 4 June 2018 at 20:06, Tomas Kalibera wrote:
> | thanks for the report. Access to the test system is not necessary, the
> | memory requirements of the byte-code compiler are usually
> | platform-independent and specifically with this package I can reproduce
> | they are very high. We'll have a look what we can do, certainly there
> | should at least be a way to recover and use the uncompiled version when
> | memory allocation fails, this is already done by the JIT but not when
> | compiling during installation. Before R or the package is patched, the
> | only workaround for memory constrained systems is probably to disable
> | byte-compilation of this package, as I read you are doing already.
>
> Yes. And as a shortcut, we just turned it off unconditionally, ie on all
> build architectures.  Worked fine as per
>
>     https://buildd.debian.org/status/package.php?p=fbasics
>
> it has been built everywhere where we have R 3.5.0 (some 20 or so platforms).
>
> The fix you suggest sounds ideal: if possible recover, and maybe WARN.
>
> Dirk
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel