parallelize Own Package in R

Question

As recommended in other posts I wrote my own package in R to parallelize functions I wrote with Rcpp. I can load the package and everything works, but when I'm using optimParallel, I get the message:

Error in checkForRemoteErrors(val) : 3 nodes produced errors; first error: object '_EffES_profileLLcpp' not found

Here is what I'm doing:

library(optimParallel)
library(EffES) # EffES is my own package

cl <- makeCluster(detectCores()-1)
clusterEvalQ(cl, library(EffES))
clusterEvalQ(cl, library(optimParallel))
setDefaultCluster(cl = cl)

w.es <- optimParallel(par=rep(0.001,3), profileLLcpp, y=y.test, x=x.test, lower = rep(0.001,3), method = "L-BFGS-B")$par

Error in checkForRemoteErrors(val) : 
  3 nodes produced errors; first error: object '_EffES_profileLLcpp' not found

What am I doing wrong?

Answer 1

You have to spread the object '_EffES_profileLLcpp' to each core of your cluster. You can do this using clusterExport , in your case:

clusterExport(cl,'_EffES_profileLLcpp')

Repeat this step with every object needed to be used in parallel (or just check which object shows up in the error log and spreat it using clusterExport ).

Hope this helps

Answer 2

Edit: The problem is solved in optimParallel version 0.7-4

The version is available on CRAN: https://CRAN.R-project.org/package=optimParallel

---

For older versions:

As detailed in this post optimParallel() needs to trick a bit in order to have no restrictions on the argument names that can be passed through the ... argument. Currently, this implies that the function passed to optimParallel() has to be defined in the .GlobalEnv in order to find compiled code properly.

Hence, a workaround could be to define the function in the .GlobalEnv :

library(optimParallel)
library(EffES)                          # EffES is your own package
cl <- makeCluster(detectCores()-1)
clusterEvalQ(cl, library(EffES))
setDefaultCluster(cl=cl)

f <- function(par, y, x) {
    profileLLcpp(par=par, x=x, y=y)
}
optimParallel(par=rep(0.001,3), f=f, y=y.test, x=x.test, 
              lower = rep(0.001,3), method = "L-BFGS-B")$par

Suggestions to improve the code of optimParallel() are welcome. I opened a corresponding question here .