does parLapplyLB do load-balancing?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

does parLapplyLB do load-balancing?

Chris Paciorek
Looking at parLapplyLB, one sees that it takes in X and then passes
splitList(X, length(cl)) to clusterApplyLB, which then calls
dynamicClusterApply.  Thus while dynamicClusterApply does handle tasks
in a load-balancing fashion, sending out individual tasks as previous
tasks complete, parLapplyLB preempts that by splitting up the tasks in
advance into as many groups of tasks as there are cluster processes.
This seems to defeat the purpose of load-balancing and of the manner
in which dynamicClusterApply is coded.

This question basically repeats a question posed in 2013 -- see
http://r.789695.n4.nabble.com/parLapplyLB-Load-balancing-tt4671848.html

I'm reposting because there doesn't seem to have been any response to
the previous posting, and it looks like the issue is still present in
R 3.1 so it seems asking again if this is the intended behavior of
parLapplyLB.

I'm using R 3.1.1 and the 3.1.1 version of the parallel package under
Ubuntu 14.04, but the code appears to be the same in R-devel.


Chris


----------------------------------------------------------------------------------------------
Chris Paciorek

Statistical Computing Consultant
Statistical Computing Facility, Econometrics Laboratory, Berkeley
Research Computing

Office: 495 Evans Hall                      Email: [hidden email]
Mailing Address:                            Voice: 510-842-6670
Department of Statistics                    Fax:   510-642-7892
367 Evans Hall                              Skype: cjpaciorek
University of California, Berkeley          WWW:
www.stat.berkeley.edu/~paciorek
Berkeley, CA 94720 USA                      Permanent forward:
[hidden email]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Reply | Threaded
Open this post in threaded view
|

Re: does parLapplyLB do load-balancing?

Ei-ji Nakama
Hello,

In such cases, try the Rhpc package.

The following is the result of the benchmark.
http://prs.ism.ac.jp/~nakama/Rhpc/#benchmark

# but tuning is not finished...


2014-12-06 10:36 GMT+09:00 Chris Paciorek <[hidden email]>:

> Looking at parLapplyLB, one sees that it takes in X and then passes
> splitList(X, length(cl)) to clusterApplyLB, which then calls
> dynamicClusterApply.  Thus while dynamicClusterApply does handle tasks
> in a load-balancing fashion, sending out individual tasks as previous
> tasks complete, parLapplyLB preempts that by splitting up the tasks in
> advance into as many groups of tasks as there are cluster processes.
> This seems to defeat the purpose of load-balancing and of the manner
> in which dynamicClusterApply is coded.
>
> This question basically repeats a question posed in 2013 -- see
> http://r.789695.n4.nabble.com/parLapplyLB-Load-balancing-tt4671848.html
>
> I'm reposting because there doesn't seem to have been any response to
> the previous posting, and it looks like the issue is still present in
> R 3.1 so it seems asking again if this is the intended behavior of
> parLapplyLB.
>
> I'm using R 3.1.1 and the 3.1.1 version of the parallel package under
> Ubuntu 14.04, but the code appears to be the same in R-devel.
>
>
> Chris
>
>
> ----------------------------------------------------------------------------------------------
> Chris Paciorek
>
> Statistical Computing Consultant
> Statistical Computing Facility, Econometrics Laboratory, Berkeley
> Research Computing
>
> Office: 495 Evans Hall                      Email: [hidden email]
> Mailing Address:                            Voice: 510-842-6670
> Department of Statistics                    Fax:   510-642-7892
> 367 Evans Hall                              Skype: cjpaciorek
> University of California, Berkeley          WWW:
> www.stat.berkeley.edu/~paciorek
> Berkeley, CA 94720 USA                      Permanent forward:
> [hidden email]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



--
Best Regards,
--
Eiji NAKAMA <nakama (a) ki.rim.or.jp>
"\u4e2d\u9593\u6804\u6cbb"  <nakama (a) ki.rim.or.jp>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel