scheduled cores … did not deliver results, all values of the jobs will be affected in parallel::mclapply() in R 4.0.1

Question

I'm using parallel::mclapply() with R 4.0.1 and getting the following warning: " scheduled cores... did not deliver results, all values of the jobs will be affected ".

Here the result of my investigation: inspecting the function source code, I realized that it happens when the vector dr is not all TRUE. This means that for some cores the second condition inside the for loop below ( is.raw(a) ) is never executed. a is the value returned by readChild() , that if returned raw data at least once, the condition would be verified at least once. So I'm thinking that readChild() is returning NULL.

readChild and readChildren return a raw vector with a "pid" attribute if data were available, an integer vector of length one with the process ID if a child terminated or NULL if the child no longer exists (no children at all for readChildren).

I ask you to validate or reject my conclusions. Finally if true what are the possible reasons?

    while (!all(fin)) {
        s <- selectChildren(ac[!fin], -1)
        if (is.null(s)) break # no children -> no hope we get anything (should not happen)
        if (is.integer(s))
            for (ch in s) {
                a <- readChild(ch)
                if (is.integer(a)) {
                    core <- which(cp == a)
                    fin[core] <- TRUE
                } else if (is.raw(a)) {
                    core <- which(cp == attr(a, "pid"))
                    job.res[[core]] <- ijr <- unserialize(a)
                    if (inherits(ijr, "try-error"))
                        has.errors <- c(has.errors, core)
                    dr[core] <- TRUE
                } else if (is.null(a)) {
                    # the child no longer exists (should not happen)
                    core <- which(cp == ch)
                    fin[core] <- TRUE
                }
            }
    }

Answer 1

This error message can occur when the child process dies/crashes, eg

> y <- parallel::mclapply(1:2, FUN = function(x) if (x == 1) quit("no") else x)
Warning message:
In parallel::mclapply(1:2, FUN = function(x) if (x == 1) quit("no") else x) :
  scheduled core 1 did not deliver a result, all values of the job will be affected

> str(y)
List of 2
 $ : NULL
 $ : int 2

That a child process completely dies is of course not good. It can happen for several reasons. My best guess is that you parallelize something that must not be parallelized. Forked processing (= mclapply() ) is known to be unstable with code that multi-thread, among other things.

For what's it worth, if you use the future framework instead (disclaimer: I'm the author), you'll get an error message that is a bit more informative, eg

> library(future.apply)
> plan(multicore)

> y <- future_lapply(1:2, FUN = function(x) if (x == 1) quit("no") else x)
Error: Failed to retrieve the result of MulticoreFuture (future_lapply-1) from
the forked worker (on localhost; PID 19959). Post-mortem diagnostic: No process
exists with this PID, i.e. the forked localhost worker is no longer alive.