Python 多处理始终使用 x 核数

Question

I have the following code that uses python's multiprocessing library to do some heavy computation using 8 cores.

import random
from multiprocessing import Process
random_numbers_list = [random.random()] * 10000000 

for i in range(0, len(random_numbers_list), 8):
    threads = []
    for j in range(i, i + 8):
        if j > len(random_numbers_list) - 1:
            break
        p = Process(target=do_heavy_computation,
                    args=(random_numbers_list[j], j))
        threads.append(p)

    [t.start() for t in threads]
    [t.join() for t in threads]

However, the problem is, while all 8 cores is correctly utilized to do the computation in parallel, it waits for the current 8 cores batch to finish until the next 8 cores batch starts. I would like the code to always utilize 8 cores, so that when any of the cores are done, it uses the next available core out of the total 8 cores. Any guidance would be greatly appreciated!

Answer 1

Let a process pool take care of everything for you.

from multiprocessing import Pool

with Pool(processes=8) as pool:
    for index, value in enumerate(random_numbers_list):
        pool.apply(do_heavy_computation, args=(value, index))

The Pool() initializer has additional arguments that you can investigate. But the above code is all you need to run your code on 8 processes, and wait until it's finished.

The most important argument you may need to change is maxtasksperchild , whose default value is None . Eight threads will be created and all your tasks will be run on those eight threads. You can change this to masktasksperchild=1 , and each thread will run only 1 task, and then die and be replaced by a fresh thread. That's what you're doing above, but it's unclear if that's necessary.