如何迫使闲置的工人并行工作？

Question

I am new to posting here--I searched and couldn't find an answer to my question. I have run the following R parallelized code (from a blog on parallel computing in R) using the parallel package on two different machines and yet get very different process time results. The first machine is a Lenovo laptop with Windows 8, 8GB RAM, Intel i7, 2 cores/4 logical processors. The second machine is a Dell desktop, Windows 7, 16GB RAM, Intel i7, 4 cores/8 logical processors. The code sometimes runs much slower on the second machine. I believe the reason is that the second machine is not using the worker nodes to complete the task. When I use the function snow.time() from the snow package to check node usage, the first machine is using all available workers to complete the task. However, on the more powerful machine, it never uses the workers--the entire task is handled by the master. Why is the first machine using workers, but the second is not with the exact same code? And how do I 'force' the second machine to use the available workers so that the code is truly parallelized and the processing time is sped up? The answers to these would help me tremendously with other work I am doing. Thanks in advance. The graphs from the function snow.time() are below as well as the code I used:

runs <- 1e7
manyruns <- function(n) mean(unlist(lapply(X=1:(runs/4), FUN=onerun)))

library(parallel)
cores <- 4
cl <- makeCluster(cores)

# Send function to workers
tobeignored <- clusterEvalQ(cl, {
    onerun <- function(.){ # Function of no arguments
        doors <- 1:3
        prize.door <- sample(doors, size=1)
        choice <- sample(doors, size=1)
        if (choice==prize.door) return(0) else return(1) # Always switch
    }
    ; NULL
})

# Send runs to the workers
tobeignored <- clusterEvalQ(cl, {runs <- 1e7; NULL})
runtime <- snow.time(avg <- mean(unlist(clusterApply(cl=cl, x=rep(runs, 4), fun=manyruns))))
stopCluster(cl)

plot(runtime)

Answer 1

Try clusterApplyLB instead of clusterApply . The "LB" is for load balancing.

The non LB version divides the number of tasks between the nodes and sends them in a batch, but if one node finishes early then it sits idle waiting for the others.

The LB version sends one task to each node then watches the nodes and when a node finishes it sends another task to that node until all the tasks are assigned. This is more efficient if the time for each task varies widely, but is less efficient if all the tasks will take about the same amount of time.

Also check the versions of R and parallel. If I am remembering correctly the clusterApply function used to not do things in parallel on Windows machines (but I don't see that note any more, so that has likely been remedied in recent versions), so the difference could be different versions of the parallel package. The parLapply function did not have the same issue, so you could rewrite your code to use it instead and see if that makes a difference.

Answer 2

I don't think it's possible to use the snow.timing function from the snow package while getting all of the other functions from the parallel package. The source for parallel in R 3.2.3 has some place holder code for timing, but it doesn't appear to be either complete or compatible with the snow.timing function in snow. I think you'll still get correct results from clusterApply , but the object returned by snow.time will be equivalent to executing:

runtime <- snow.time(Sys.sleep(20))

If you want to use snow.timing , I suggest only loading snow, although you can still access functions such as detectCores using the syntax parallel::detectCores() .

I don't really know why your script occasionally runs slowly on your desktop machine, but I think that the way you are parallelizing it is reasonable and correct. You might want to try benchmarking manyruns sequentially on both machines in order to rule out any differences in the random number generation code on the two systems. But perhaps the problem was caused by a system service that was slowing down the whole system.

Answer 3

I cannot put code in comments... I do no understand your program very well. What kind of cluster are you creating? Try this, adjust 2e6 to whatever works for you:

library(parallel)
library(Rmpi)
library(snow)
cl <- makeMPIcluster(3)
t <- system.time(parLapply(cl, 1:100, function(i) mean(rnorm(2e6))))
stopCluster(cl)
print(t)

for me it runs for 10 seconds (2 core/hyperthreading/5y old laptop/linux), all 4 workers are 100% busy. You may also try the same with socket clusters.