undefined S4 class in parallel computing at snowfall

[Edwin Sun]

Dear All,

I have a question of how to export S4 class specification to clusters/workers in parallel computing. The package I used is snowfall. The problem is reproducible as follows. Any hint is greatly appreciated.

Edwin Sun

=== begin ===========
library(snowfall)
sfInit(parallel = TRUE, cpus = 2)
setClass("catt", representation(aa = "numeric"))
setClass("dogg", representation(bb = 'character', cc = "catt"))
f1 <- function(y1) { new('catt', aa = y1 + 1) }
f3 <- function(y2, y3) { new('dogg', bb=c(y2, 'GA'), cc = f1(y3)) }
dat <- 1:5
f1(dat)
f3('NY', dat)    
sapply(c("state1", 'state2', 'state3'), f3, dat)
sfExportAll()
sfClusterEval(ls())
sfSapply(c("s1", 's2', 's3'), f3, dat)
sfStop()

# sfSapply generates the following error
#    Error in checkForRemoteErrors(val) :
#      2 nodes produced errors; first error: "dogg" is not a defined class
=== end =============

[Martin Morgan]

Hi --

On 06/28/2012 04:51 PM, Edwin Sun wrote:

> Dear All,
>
> I have a question of how to export S4 class specification to
> clusters/workers in parallel computing. The package I used is snowfall. The
> problem is reproducible as follows. Any hint is greatly appreciated.
>
> Edwin Sun
>
> === begin ===========
> library(snowfall)
> sfInit(parallel = TRUE, cpus = 2)
> setClass("catt", representation(aa = "numeric"))
> setClass("dogg", representation(bb = 'character', cc = "catt"))
> f1<- function(y1) { new('catt', aa = y1 + 1) }
> f3<- function(y2, y3) { new('dogg', bb=c(y2, 'GA'), cc = f1(y3)) }
> dat<- 1:5
> f1(dat)
> f3('NY', dat)
> sapply(c("state1", 'state2', 'state3'), f3, dat)
> sfExportAll()

I think it makes sense to actually define the classes (and other
cluster-specific objects) on the cluster;

   sfInit(parallel = TRUE, cpus = 2)
   sfClusterEval({
       setClass("catt", representation(aa = "numeric"))
       setClass("dogg", representation(bb = 'character', cc = "catt"))
       f1 <- function(y1) new('catt', aa = y1 + 1)
   })

   f3 <- function(y2, y3) { new('dogg', bb=c(y2, 'GA'), cc = f1(y3)) }
   dat <- 1:5
   sfSapply(c("s1", 's2', 's3'), f3, dat)

It's true that setClass creates variables in the global environment (see
the .__C__* objects in the output of ls(all=TRUE)) and that sfExportAll
seems not to export these, so that you could export them explicitly,
e.g., sfExport(list=ls(all=TRUE)), but this assuming that these
variables are the only side effect of calling setClass. Maybe the
'setup' actions are defined in a separate script setup.R that
sfClusterEval (and the main program, if that's relevant) source,
sfClusterEval(source("setup.R")).

Martin

> sfClusterEval(ls())
> sfSapply(c("s1", 's2', 's3'), f3, dat)
> sfStop()
>
> # sfSapply generates the following error
> #    Error in checkForRemoteErrors(val) :
> #      2 nodes produced errors; first error: "dogg" is not a defined class
> === end =============
>
> --
> View this message in context: http://r.789695.n4.nabble.com/undefined-S4-class-in-parallel-computing-at-snowfall-tp4634757.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: [phone]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.