Is it possible, in R parallel::mcparallel, to limit the number of cores used at any one time?

Question

In R, the mcparallel() function in the parallel package forks off a new task to a worker each time it is called. If my machine has N (physical) cores, and I fork off 2N tasks, for example, then each core starts off running two tasks, which is not desirable. I would rather like to be able to start running N tasks on N workers, and then, as each tasks finishes, submit the next task to the now-available core. Is there an easy way to do this?

My tasks take different amounts of time, so it is not an option to fork off the tasks serial in batches of N. There might be some workarounds, such as checking the number of active cores and then submitting new tasks when they become free, but does anyone know of a simple solution?

I have tried setting cl <- makeForkCluster(nnodes=N) , which does indeed set N cores going, but these are not then used by mcparallel() . Indeed, there appears to be no way to feed cl into mcparallel() . The latter has an option mc.affinity , but it's unclear how to use this and it doesn't seem to do what I want anyway (and according to the documentation its functionality is machine dependent).

Answer 1

you have at least 2 possibilities:

1. As mentioned above you can use mcparallel's parameters "mc.cores" or "mc.affinity". On AMD platforms "mc.affinity" is preferred since two cores share same clock. For example an FX-8350 has 8 cores, but core 0 has same clock as core 1. If you start a task for 2 cores only it is better to assign it to cores 0 and 1 rather than 0 and 2. "mc.affinity" makes that. The price is loosing load balancing.

   "mc.affinity" is present in recent versions of the package. See changelog to find when introduced.

2. Also you can use OS's tool for setting affinity, eg "taskset":

   /usr/bin/taskset -c 0-1 /usr/bin/R ...

   Here you make your script to run on cores 0 and 1 only.

Keep in mind Linux numbers its cores starting from "0". Package parallel conforms to R's indexing and first core is core number 1.

Answer 2

I'd suggest taking advantage of the higher level functions in parallel that include this functionality instead of trying to force low level functions to do what you want.

In this case, try writing your tasks as different arguments of a single function. Then you can use mclapply() with the mc.preschedule parameter set to TRUE and the mc.cores parameter set to the number of threads you want to use at a time. Each time a task finishes and a thread closes, a new thread will be created, operating on the next available task.

Even if each task uses a completely different bit of code, you can create a list of functions and pass that to a wrapper function. For example, the following code executes two functions at a time.

f1 <- function(x) {x^2}
f2 <- function(x) {2*x}
f3 <- function(x) {3*x}
f4 <- function(x) {x*3}
params <- list(f1,f2,f3,f4)
wrapper <- function(f,inx){f(inx)}
output <- mclapply(params,FUN=calling,mc.preschedule=TRUE,mc.cores=2,inx=5)

If need be you could make params a list of lists including various parameters to be passed to each function as well as the function definition. I've used this approach frequently with various tasks of different lengths and it works well.

Of course, it may be that your various tasks are just different calls to the same function, in which case you can use mclapply directly without having to write a wrapper function.