parallel computing with foreach()

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

parallel computing with foreach()

Kumar Mainali
I have used foreach() for parallel computing but in the current problem, it
is not working. Given the volume and type of the data involved in the
analysis, I will try to give below the complete code without reproducible
example.

In short, each R environment will draw a set of separate files, perform the
analysis and dump in separate folders.

splist <- c("juoc", "juos", "jusc", "pico", "pifl", "pipo", "pire", "psme")
covset <- c("PEN", "Thorn")

foreach(i = 1:length(splist)) %:%
foreach(j = 1:length(covset)) %dopar% {

spname <- splist[i]; spname
myTorP <- covset[j]; myTorP

DataSpecies = data.frame(prsabs = rep(1, 10), lon = rep(30, 10), lat =
rep(80, 10))
myResp = as.numeric(DataSpecies[,1])
myRespXY = DataSpecies[, c("lon", "lat")]
# directory of a bunch of raster files specific to each R environment
rastdir <- paste0(rootdir, "Current/", myTorP); rastdir
rasterc = list.files(rastdir, pattern="\\.tif$", full.names = T)
print(rasterc)
myExplc = stack(rasterc, RAT=FALSE)
}

I get the following error message that most likely generates while stacking
rasters because there are 25 rasters in the folder of each environment.
Also, in the normal for loop, this reads all fine.
Error in { :
  task 1 failed - "arguments imply differing number of rows: 25, 0"

Thank you.


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: parallel computing with foreach()

plangfelder
Your code generates an error that has nothing to do with dopar. I have
no idea what your function stack is supposed to do; you may be
inadvertently calling utils::stack which would produce this kind of
error:

> stack(1:25, RAT = FALSE)
Error in data.frame(values = unlist(unname(x)), ind, stringsAsFactors = FALSE) :
  arguments imply differing number of rows: 25, 0

HTH,

Peter

On Wed, Dec 6, 2017 at 10:03 PM, Kumar Mainali <[hidden email]> wrote:

> I have used foreach() for parallel computing but in the current problem, it
> is not working. Given the volume and type of the data involved in the
> analysis, I will try to give below the complete code without reproducible
> example.
>
> In short, each R environment will draw a set of separate files, perform the
> analysis and dump in separate folders.
>
> splist <- c("juoc", "juos", "jusc", "pico", "pifl", "pipo", "pire", "psme")
> covset <- c("PEN", "Thorn")
>
> foreach(i = 1:length(splist)) %:%
> foreach(j = 1:length(covset)) %dopar% {
>
> spname <- splist[i]; spname
> myTorP <- covset[j]; myTorP
>
> DataSpecies = data.frame(prsabs = rep(1, 10), lon = rep(30, 10), lat =
> rep(80, 10))
> myResp = as.numeric(DataSpecies[,1])
> myRespXY = DataSpecies[, c("lon", "lat")]
> # directory of a bunch of raster files specific to each R environment
> rastdir <- paste0(rootdir, "Current/", myTorP); rastdir
> rasterc = list.files(rastdir, pattern="\\.tif$", full.names = T)
> print(rasterc)
> myExplc = stack(rasterc, RAT=FALSE)
> }
>
> I get the following error message that most likely generates while stacking
> rasters because there are 25 rasters in the folder of each environment.
> Also, in the normal for loop, this reads all fine.
> Error in { :
>   task 1 failed - "arguments imply differing number of rows: 25, 0"
>
> Thank you.
> ᐧ
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: parallel computing with foreach()

Kumar Mainali
Thanks Peter. I failed to realize earlier that one of the functions I used
came from a package. The following solved the problem.

foreach(i = 1:length(splist)) %:%

foreach(j = 1:length(covset), .packages = c("raster")) %dopar% {

.......


On Thu, Dec 7, 2017 at 1:52 AM, Peter Langfelder <[hidden email]
> wrote:

> Your code generates an error that has nothing to do with dopar. I have
> no idea what your function stack is supposed to do; you may be
> inadvertently calling utils::stack which would produce this kind of
> error:
>
> > stack(1:25, RAT = FALSE)
> Error in data.frame(values = unlist(unname(x)), ind, stringsAsFactors =
> FALSE) :
>   arguments imply differing number of rows: 25, 0
>
> HTH,
>
> Peter
>
> On Wed, Dec 6, 2017 at 10:03 PM, Kumar Mainali <[hidden email]>
> wrote:
> > I have used foreach() for parallel computing but in the current problem,
> it
> > is not working. Given the volume and type of the data involved in the
> > analysis, I will try to give below the complete code without reproducible
> > example.
> >
> > In short, each R environment will draw a set of separate files, perform
> the
> > analysis and dump in separate folders.
> >
> > splist <- c("juoc", "juos", "jusc", "pico", "pifl", "pipo", "pire",
> "psme")
> > covset <- c("PEN", "Thorn")
> >
> > foreach(i = 1:length(splist)) %:%
> > foreach(j = 1:length(covset)) %dopar% {
> >
> > spname <- splist[i]; spname
> > myTorP <- covset[j]; myTorP
> >
> > DataSpecies = data.frame(prsabs = rep(1, 10), lon = rep(30, 10), lat =
> > rep(80, 10))
> > myResp = as.numeric(DataSpecies[,1])
> > myRespXY = DataSpecies[, c("lon", "lat")]
> > # directory of a bunch of raster files specific to each R environment
> > rastdir <- paste0(rootdir, "Current/", myTorP); rastdir
> > rasterc = list.files(rastdir, pattern="\\.tif$", full.names = T)
> > print(rasterc)
> > myExplc = stack(rasterc, RAT=FALSE)
> > }
> >
> > I get the following error message that most likely generates while
> stacking
> > rasters because there are 25 rasters in the folder of each environment.
> > Also, in the normal for loop, this reads all fine.
> > Error in { :
> >   task 1 failed - "arguments imply differing number of rows: 25, 0"
> >
> > Thank you.
> > ᐧ
> >
> >         [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/
> posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>



--
Postdoctoral Associate
Department of Biology
University of Maryland, College Park


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.