R 3.4 and mclapply assertion failure - is this a bug?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

R 3.4 and mclapply assertion failure - is this a bug?

R devel mailing list
Dear all, 
I am not sure if this is a bug, so I prefer to post it here before filing.
After upgrading to 3.4 I encounter the following message with mclapply:
Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
The complete pdf of what console produces is here:https://www.dropbox.com/s/navugg28gqne14p/R%20Console.pdf?dl=0
The batch with commands to (hopefully) reproduce it is here:https://www.dropbox.com/s/8pd0ij3kowigdlc/20170430%20-%20R%20mclapply%20bug%20report.r?dl=0
The data for the reproducible example is here:
https://www.dropbox.com/s/2enoeapu7jgcxwd/z.Rdata?dl=0
Note that this behavior was not present in R 3.3.3 on the same machine and all the batch was working without any errors.
The sessionInfo():
R version 3.4.0 (2017-04-21)Platform: x86_64-apple-darwin15.6.0 (64-bit)Running under: macOS Sierra 10.12.4
Matrix products: defaultBLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylibLAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
locale:[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:[1] parallel  compiler  stats     graphics  grDevices utils     datasets  methods   base     
other attached packages: [1] data.table_1.10.4      numbers_0.6-6          microbenchmark_1.4-2.1 zoo_1.8-0              doParallel_1.0.10      iterators_1.0.8        [7] foreach_1.4.3          RSclient_0.7-3         stringi_1.1.5          stringr_1.2.0          lubridate_1.6.0        plyr_1.8.4            
loaded via a namespace (and not attached): [1] Rcpp_0.12.10     lattice_0.20-35  codetools_0.2-15 grid_3.4.0       gtable_0.2.0     magrittr_1.5     scales_0.4.1     ggplot2_2.2.1    [9] lazyeval_0.2.0   tools_3.4.0      munsell_0.4.3    colorspace_1.3-2 tibble_1.3.0    

Thank you in advance for help/hints, Yan
        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R 3.4 and mclapply assertion failure - is this a bug?

Simon Urbanek
Yan,

this looks like an interaction of OpenMP in data.table and forking - you may want to report that to Matt as he can shed more light on that. He has been complaining that we didn't enable OpenMP before 3.4 so he probably knows about the issue and how to fix it.

Cheers,
Simon


> On May 2, 2017, at 7:12 AM, Yan Alperovych via R-devel <[hidden email]> wrote:
>
> Dear all,
> I am not sure if this is a bug, so I prefer to post it here before filing.
> After upgrading to 3.4 I encounter the following message with mclapply:
> Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
> The complete pdf of what console produces is here:https://www.dropbox.com/s/navugg28gqne14p/R%20Console.pdf?dl=0
> The batch with commands to (hopefully) reproduce it is here:https://www.dropbox.com/s/8pd0ij3kowigdlc/20170430%20-%20R%20mclapply%20bug%20report.r?dl=0
> The data for the reproducible example is here:
> https://www.dropbox.com/s/2enoeapu7jgcxwd/z.Rdata?dl=0
> Note that this behavior was not present in R 3.3.3 on the same machine and all the batch was working without any errors.
> The sessionInfo():
> R version 3.4.0 (2017-04-21)Platform: x86_64-apple-darwin15.6.0 (64-bit)Running under: macOS Sierra 10.12.4
> Matrix products: defaultBLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylibLAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
> locale:[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
> attached base packages:[1] parallel  compiler  stats     graphics  grDevices utils     datasets  methods   base    
> other attached packages: [1] data.table_1.10.4      numbers_0.6-6          microbenchmark_1.4-2.1 zoo_1.8-0              doParallel_1.0.10      iterators_1.0.8        [7] foreach_1.4.3          RSclient_0.7-3         stringi_1.1.5          stringr_1.2.0          lubridate_1.6.0        plyr_1.8.4            
> loaded via a namespace (and not attached): [1] Rcpp_0.12.10     lattice_0.20-35  codetools_0.2-15 grid_3.4.0       gtable_0.2.0     magrittr_1.5     scales_0.4.1     ggplot2_2.2.1    [9] lazyeval_0.2.0   tools_3.4.0      munsell_0.4.3    colorspace_1.3-2 tibble_1.3.0    
>
> Thank you in advance for help/hints, Yan
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: R 3.4 and mclapply assertion failure - is this a bug?

Matt Dowle
In reply to this post by R devel mailing list
That's great news that OpenMP is now enabled in MacOS CRAN binaries! Thanks
Simon.

data.table is supposed to automatically switch down to single-threaded mode
when explicitly parallelized via the fork mechanism.   It does that via
functions registered in init.c as follows :

    pthread_atfork(&when_fork, &when_fork_end, NULL)

Tests 1705 and 1706 test this is working.   Those tests are passing ok on
CRAN currently including x86_64-apple-darwin15.6.0.

I'll work with Yan off list to see what the difference is.  Either the fork
catch isn't working in that environment for some reason, the use case is
different to the tests somehow or the fork catch is working but something
somewhere is asserting its surprise at being single threaded.  In the
meantime setDTthreads(1) can be called manually before the explicit
parallelism as a workaround.

Matt

On Tue, May 2, 2017 at 1:25 PM, Yan Alperovych <[hidden email]>
wrote:

> Dear Matt,
>
> I follow the Simon’s advice on this. Since R 3.4 the mclapply is producing
> a strange error which was not present in R 3.3.3. Simon’s answer is below
> and the relevant info (the console output, the code, and the sample data)
> are all below.
>
> Thank you very much for helping,
> Yan
>
> Begin forwarded message:
>
> *From: *Simon Urbanek <[hidden email]>
> *Subject: **Re: [Rd] R 3.4 and mclapply assertion failure - is this a
> bug?*
> *Date: *May 2, 2017 at 7:09:36 PM GMT+2
> *To: *Yan Alperovych <[hidden email]>
> *Cc: *"[hidden email]" <[hidden email]>
>
> Yan,
>
> this looks like an interaction of OpenMP in data.table and forking - you
> may want to report that to Matt as he can shed more light on that. He has
> been complaining that we didn't enable OpenMP before 3.4 so he probably
> knows about the issue and how to fix it.
>
> Cheers,
> Simon
>
>
> On May 2, 2017, at 7:12 AM, Yan Alperovych via R-devel <
> [hidden email]> wrote:
>
> Dear all,
>
>
> I am not sure if this is a bug, so I prefer to post it here before filing.
>
>
> After upgrading to 3.4 I encounter the following message with mclapply:
>
>
> Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool ==
> __null.OMP : Error #13: Assertion failure at kmp_runtime.cpp(6480).OMP:
> Hint: Please submit a bug report with this message, compile and run
> commands used, and machine configuration info including native compiler and
> operating system versions. Faster response will be obtained by including
> all program sources. For information on submitting this issue, please see
> http://www.intel.com/software/products/support/.
>
>
> The complete pdf of what console produces is here:
>
> https://www.dropbox.com/s/navugg28gqne14p/R%20Console.pdf?dl=0
>
>
> The batch with commands to (hopefully) reproduce it is here:
>
> <a href="https://www.dropbox.com/s/8pd0ij3kowigdlc/20170430%20-%">https://www.dropbox.com/s/8pd0ij3kowigdlc/20170430%20-%
> 20R%20mclapply%20bug%20report.r?dl=0
>
>
> The data for the reproducible example is here:
> https://www.dropbox.com/s/2enoeapu7jgcxwd/z.Rdata?dl=0
>
>
> Note that this behavior was not present in R 3.3.3 on the same machine and
> all the batch was working without any errors.
>
>
> The sessionInfo():
> R version 3.4.0 (2017-04-21)
>
> Platform: x86_64-apple-darwin15.6.0 (64-bit)
>
> Running under: macOS Sierra 10.12.4
>
>
> Matrix products: default
>
>
> BLAS: /Library/Frameworks/R.framework/Versions/3.4/
> Resources/lib/libRblas.0.dylib
>
> LAPACK: /Library/Frameworks/R.framework/Versions/3.4/
> Resources/lib/libRlapack.dylib
>
>
> *locale:*
> *[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8*
>
> *attached base packages:*
> *[1] parallel  compiler  stats     graphics  grDevices utils     datasets
>  methods   base     *
>
> *other attached packages:*
> * [1] data.table_1.10.4      numbers_0.6-6          microbenchmark_1.4-2.1
> zoo_1.8-0              doParallel_1.0.10      iterators_1.0.8       *
> * [7] foreach_1.4.3          RSclient_0.7-3         stringi_1.1.5
>  stringr_1.2.0          lubridate_1.6.0        plyr_1.8.4            *
>
> *loaded via a namespace (and not attached):*
> * [1] Rcpp_0.12.10     lattice_0.20-35  codetools_0.2-15 grid_3.4.0
> gtable_0.2.0     magrittr_1.5     scales_0.4.1     ggplot2_2.2.1   *
> * [9] lazyeval_0.2.0   tools_3.4.0      munsell_0.4.3    colorspace_1.3-2
> tibble_1.3.0    *
>
>
>
> Thank you in advance for help/hints, Yan
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel