When (if ever) should I tell R parallel to not use all cores?

Question

I've been using this code:

library(parallel)
cl <- makeCluster( detectCores() - 1)
clusterCall(cl, function(){library(imager)})

then I have a wrapper function looking something like this:

d <- matrix  #Loading a batch of data into a matrix
res <- parApply(cl, d, 1, FUN, ...)
# Upload `res` somewhere

I tested on my notebook, with 8 cores (4 cores, hyperthreading). When I ran it on a 50,000 row, 800 column, matrix, it took 177.5s to complete, and for most of the time the 7 cores were kept at near 100% (according to top), then it sat there for the last 15 or so seconds, which I guess was combining results. According to system.time() , user time was 14s, so that matches.

Now I'm running on EC2, a 36-core c4.8xlarge, and I'm seeing it spending almost all of its time with just one core at 100%. More precisely: There is an approx 10-20 secs burst where all cores are being used, then about 90 secs of just one core at 100% (being used by R), then about 45 secs of other stuff (where I save results and load the next batch of data). I'm doing batches of 40,000 rows, 800 columns.

The long-term load average, according to top, is hovering around 5.00.

Does this seem reasonable? Or is there a point where R parallelism spends more time with communication overhead, and I should be limiting to eg 16 cores. Any rules of thumb here?

Ref: CPU spec I'm using "Linux 4.4.5-15.26.amzn1.x86_64 (amd64)". R version 3.2.2 (2015-08-14)

UPDATE: I tried with 16 cores. For the smallest data, run-time increased from 13.9s to 18.3s. For the medium-sized data:

With 16 cores:
   user  system elapsed 
 30.424   0.580  60.034 

With 35 cores:
   user  system elapsed 
 30.220   0.604  54.395 

Ie the overhead part took the same amount of time, but the parallel bit had fewer cores so took longer, and so it took longer overall.

I also tried using mclapply() , as suggested in the comments. It did appear to be a bit quicker (something like 330s vs. 360s on the particular test data I tried it on), but that was on my notebook, where other processes, or over-heating, could affect the results. So, I'm not drawing any conclusions on that yet.

Answer 1

There are no useful rules of thumb — the number of cores that a parallel task is optimal for is entirely determined by said task. For a more general discussion see Gustafson's law .

The high single-core portion that you're seeing in your code probably comes from the end phase of the algorithm (the “join” phase), where the parallel results are collated into a single data structure. Since this far surpasses the parallel computation phase, this may indeed be an indication that fewer cores could be beneficial.

Answer 2

I'd add that in case you are not aware of this wonderful resource for parallel computing in R, you may find reading Norman Matloff's recent book Parallel Computing for Data Science: With Examples in R, C++ and CUDA a very helpful read. I'd highly recommend it (I learnt a lot, not coming from a CS background).

The book answers your question in depth (Chapter 2 specifically). The book gives a high level overview of the causes of overhead that lead to bottlenecks to parallel programs.

Quoting section 2.1, which implicitly partially answers your question:

There are two main performance issues in parallel programming:

Communications overhead : Typically data must be transferred back and forth between processes. This takes time, which can take quite a toll on performance. In addition, the processes can get in each other's way if they all try to access the same data at once. They can collide when trying to access the same communications channel, the same memory module, and so on. This is another sap on speed. The term granularity is used to refer, roughly, to the ratio of computa- tion to overhead. Large-grained or coarse-grained algorithms involve large enough chunks of computation that the overhead isn't much of a problem. In fine-grained algorithms, we really need to avoid overhead as much as possible.

^ When overhead is high, less cores for the problem at hand can give shorter total computation time.

Load balance : As noted in the last chapter, if we are not careful in the way in which we assign work to processes, we risk assigning much more work to some than to others. This compromises performance, as it leaves some processes unproductive at the end of the run, while there is still work to be done.

When if ever do not use all cores? One example from my personal experience in running daily cronjobs in R on data that amounts to 100-200GB data in RAM, in which multiple cores are run to crunch blocks of data, I've indeed found running with say 6 out of 32 available cores to be faster than using 20-30 of the cores. A major reason was memory requirements for children processes (After a certain number of children processes were in action, memory usage got high and things slowed down considerably).