为什么多处理在Google Compute Engine中不使用100％CPU？

Question

I have to run some code in parallel and, while it uses 100% of the CPU in my local computer, it doesn't work like that in Google Compute Engine.

I'm using a Compute Engine with 24vCPUs and 22Gb memory. The following image shows the utilization of CPU while the program is running.

The mock code I'm using is:

from joblib import Parallel, delayed
import multiprocessing 

def some_function():
       num_cores = multiprocessing.cpu_count() # = 24
       salida = Parallel(n_jobs=num_cores)(delayed(hijo.calcula_error)(metodo='pond') for hijo in self.descendencia)

Where hijo.calcula_error is a function that can be run in parallel.

Is this behaviour normal or am I doing something wrong?

Answer 1

I think that it also depends very much on your code, I would review the multiprocessing methods and start testing by run it on a single core instance to see if it uses all the capacity (or maybe spawn 24 process on a 24 core instance). I hope this can help.

Other thing I would consider here is take a look on the memory bottleneck or disk I/O throttling that could be limiting the over all performance of the instance.