DEoptim error: objective function result has different length than parameter matrix due to foreachArgs specification

Question

I have a very odd DEoptim error that I have "fixed", but do not understand.

I do not have issues when I use DEoptim 's parallel capabilities with the parallel package (ie, pType=1 ). However when I use foreach instead (which I must use on the grid computing setup that is available to me), I have issues. Below is an MRE of a much simplified version of the issue that I had. pType=1 works, pType=2 when foreachArgs is specified returns an error:

objective function result has different length than parameter matrix

When I do not specify foreachArgs the issue goes away. Does anyone have thoughts about the root cause of this issue?

library(zoo)
library(parallel)
library(doParallel)
library(DEoptim)


myfunc1 <- function(params){
    s <- myfunc2(params,ncal,n_left_cens,astats, X_ret, disc_length, X_acq, POP_0, POP_ann_growth)
    loss_func(s)
    }

myfunc2 = function(params,ncal,n_left_cens,astats, X_ret, disc_length, X_acq, POP_0, POP_ann_growth){
    sum(params) + ncal + n_left_cens + astats + X_ret + disc_length + X_acq + POP_0 + POP_ann_growth 
    }

loss_func = function(s){
    s
    }   


# General setup
ncal = 1
n_left_cens = 1
astats= 1
disc_length = 1
POP_0 = 1
POP_ann_growth = 1
X_acq = 1
X_ret = 1
params = c(1,1)
W = 1

paral = TRUE
itermax=100
ncores = detectCores()
cltype <- ifelse(.Platform$OS.type != "windows", "FORK", "PSOCK")
trace=TRUE

# bounds for search for DEoptim
lower = rep(-1,length(params))
upper = lower*-1




# parallel: works               
pType = 1
parVar = c("myfunc1","myfunc2","loss_func","W","ncal","n_left_cens","astats","X_ret","disc_length",
                        "X_acq","POP_0","POP_ann_growth")               
foreachArguments <- list("myfunc1","myfunc2","loss_func","ncal","n_left_cens","astats","X_ret","disc_length",
                        "X_acq","POP_0","POP_ann_growth")
clusters <- makeCluster(ncores, type = cltype)
registerDoParallel(clusters)
clusterExport(cl=clusters, varlist=foreachArguments, envir=environment())
results <- DEoptim(fn=myfunc1,lower=lower,upper=upper,
                DEoptim.control(itermax=itermax,trace=trace,parallelType=pType,
                parVar=parVar))
showConnections(all = TRUE)
closeAllConnections()





# foreach with foreachArgs specified: doesn't work
pType = 2               
clusters <- makeCluster(ncores, type = cltype)
registerDoParallel(clusters)
clusterExport(cl=clusters, varlist=foreachArguments, envir=environment())

results <- DEoptim(fn=myfunc1,lower=lower,upper=upper,
                DEoptim.control(itermax=itermax,trace=trace,parallelType=pType,
                foreachArgs=foreachArguments))
showConnections(all = TRUE)
closeAllConnections()





# foreach with foreachArgs unspecified: works
pType = 2               
foreachArguments <- list("myfunc1","myfunc2","loss_func","ncal","n_left_cens","astats","X_ret","disc_length",
                        "X_acq","POP_0","POP_ann_growth")
clusters <- makeCluster(ncores, type = cltype)
registerDoParallel(clusters)
clusterExport(cl=clusters, varlist=foreachArguments, envir=environment())

results <- DEoptim(fn=myfunc1,lower=lower,upper=upper,
                DEoptim.control(itermax=itermax,trace=trace,parallelType=pType))
showConnections(all = TRUE)
closeAllConnections()

Answer 1

From ?DEoptim.control :

foreachArgs: A list of named arguments for the 'foreach' function from the package 'foreach'. The arguments 'i', '.combine' and '.export' are not possible to set here; they are set internally.

Which you seem to be conflating with the behavior of parVar :

parVar: Used if 'parallelType=1'; a list of variable names (as strings) that need to exist in the environment for use by the objective function or are used as arguments by the objective function.

You need to specify the arguments passed to foreach as name = value pairs. For example:

foreachArguments <- list(.export = c("myfunc1", "myfunc2", "loss_func", "ncal",
 "n_left_cens", "astats", "X_ret", "disc_length", "X_acq","POP_0","POP_ann_growth")

I'm not sure what's causing that specific error, but the fix is "don't do that." ;)

Here's an example of how you'd actually use the foreachArgs argument. Note that I'm setting the .verbose argument to make foreach print diagnostics:

library(doParallel)
library(DEoptim)
clusters <- makeCluster(detectCores())
registerDoParallel(clusters)
obj_func <- function(params) { sum(params) }
results <- DEoptim(fn=obj_func, lower=c(-1, -1), upper=c(1, 1),
  DEoptim.control(parallelType=2, foreachArgs=list(.verbose=TRUE)))
stopCluster(clusters)