计算引擎上的 Python3 多处理池不平衡 CPU 使用率

Question

I'm trying to run a Python3 program that uses multiple cores to do computations, on a Google cloud compute engine.
The code looks like this:

import multiprocessing
from multiprocessing import Pool
# functions and variables defined
MAX_PROCESS_COUNT = (multiprocessing.cpu_count() - 1) or 1
if __name__=="__main__":
    with Pool(processes=MAX_PROCESS_COUNT) as pool:
        result = list(tqdm.tqdm(pool.imap(single_task, range(len(my_list))), total=len(my_list)))

The compute engine has 20 cores, so I decide to use only 19 of them. my_list has about 200 values, and each single_task takes about 10 mins to complete on my laptop.

When I actually run the program, it took about 1.6 hours to complete only 35 tasks.
So I check the htop , and find that all CPU cores are in use, and the memory usage looks unusually low (expected 14G):

More importantly, the CPU usages for each individual task is highly unbalanced:

I believe that the unbalanced CPU usage is the problem here.
Is there any way to constrain that usage? Should I config in the VM environment or change the python code?

I've tested the same code on my laptop, and it runs as expected: out of 8 cores only 1 core is not fully utilized.

By the way, my code uses packages like NumPy, Pandas, and sklearn, and I've already set up libblas for NumPy

Answer 1

I found the solution here , which is what Klaus D. mentioned. The NumPy computations are not bound to any process, and need to to configured before running Python program.
So, in my situation, I added these lines to the top of my Python file:

import os
os.environ['MKL_NUM_THREADS'] = '1'
os.environ['NUMEXPR_NUM_THREADS'] = '1'
os.environ['OMP_NUM_THREADS'] = '1'

So that each NumPy related computation is restricted to its own assigned process.

Additionally, you can check your NumPy configuration by:

import numpy as np
np.show_config()

And see which environment variable should be set to limit the number of threads.