exiting mclapply early on error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

exiting mclapply early on error

Giovanni Righi
Hey folks,

Is there any way to exit an mclapply early on error?

For example, in the following mclapply loop, I have to wait for all the processes to finish before the error is returned.

```
mclapply(X = 1:12, FUN = function(x) {Sys.sleep(0.1); if(x == 4) stop()}, mc.cores = 4, mc.preschedule = F)
```

When there are many calculations in FUN, it takes a long time before the error is returned. It would be nice if there were an option to exit (all child processes) early on error. Is there any way to do this?

Thanks,
Giovanni Righi

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: exiting mclapply early on error

Jeremie Juste
Hello Giovanni,

I don't know if my workflow would suit you but I tend to want the
opposite when I launch a parallel process. I tend to want to keep the
processes alive as long as they can. If the computation time is long I
would not want to lose everything.

lapply..8 <- function(X,FUN,...){
    max..cores <- as.numeric(system("grep ^processor /proc/cpuinfo 2>/dev/null | wc -l",TRUE))
    mclapply(X,FUN,mc.cores=min(max..cores,8))
}

lapply_with_error..8 <- function(X,FUN,...) {    
    lapply..8(X, function(x, ...) tryCatch(FUN(x, ...),
                                            error=function(e){
                                       print(e)
                                       e}
                                       ))
}

res <- lapply_with_error..8(X = 1:12, FUN = function(x) {Sys.sleep(0.1); if(x == 4) stop()})

Then we can investigate the problem for the element that generated
errors. It is even better if we could anticipate the errors and avoid
surprises  by well _testing_ the function before launching a long
process.

If you want the processes to fail fast, I fear that you want to launch the parallel process too soon without
having tested your function enough.  

HTH,

Jeremie

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: exiting mclapply early on error

Giovanni Righi
Thanks for the response, Jeremie. I wholeheartedly agree about testing.

In my case, this feature would be used purely to reduce computation time. I’m calculating
an expensive (and embarrassingly parallel) likelihood function, and for some parameter combinations
my objective function diverges to infinity. In those instances, I’d like to throw an error that stops
subsequent computations and that I could catch to return a value of infinity. I agree, though,
that the default should allow all processes to finish.

What I’ve suggested does not seem possible with mclapply(). Do you know how I could suggest
this as a feature request to the maintainer of the parallel package?

Thanks,
Giovanni
______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: exiting mclapply early on error

Bert Gunter-2
This may be O/T. If so, either reply privately or not at all.

"objective function diverges to infinity."

How do you operationally define that? Detecting Inf -- e.g. 1/0, log(0),
etc -- is straightforward. But how do you know other than via math whether
iterations will continue to diverge or not? Would you not need to
operationalize that by defining  divergence criteria? -- e.g. too many
iterations, objective function grows/shrinks too slowly, etc. In other
words, more or less the usual adapted to your particular context.

Cheers,
Bert


Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Tue, Oct 13, 2020 at 9:13 AM Giovanni Righi <[hidden email]> wrote:

> Thanks for the response, Jeremie. I wholeheartedly agree about testing.
>
> In my case, this feature would be used purely to reduce computation time.
> I’m calculating
> an expensive (and embarrassingly parallel) likelihood function, and for
> some parameter combinations
> my objective function diverges to infinity. In those instances, I’d like
> to throw an error that stops
> subsequent computations and that I could catch to return a value of
> infinity. I agree, though,
> that the default should allow all processes to finish.
>
> What I’ve suggested does not seem possible with mclapply(). Do you know
> how I could suggest
> this as a feature request to the maintainer of the parallel package?
>
> Thanks,
> Giovanni
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.