Multiprocessing on a set number of cores

Question

In the code below, where Function is a function to be called, how might I specify the number of processors to be used as 10?

  if __name__ == '__main__':
        jobs = []

        for l in lst:
           p = multiprocessing.Process(target=Function, args=(l,))
           jobs.append(p)
           p.start()

This code will completely take over my server, so how do I limit it to ten cores? Should I put it in a loop?

Answer 1

Given that you are essentially mapping a function over a list of variables might I suggest that you use multiprocessing.Pool instead.

This is a class which creates a pool of a limited number of worker threads that can then be used to run a function over a list of inputs instead of Process where you create a thread per function call and then run them all at the same time

An example of using it in versions of python < 3.3 would be:

from multiprocessing import Pool
import contextlib

num_threads = 10

with contextlib.closing( Pool(num_threads) ) as pool:
    results = pool.map(Function, lst)

If you are using python 3 than the Pool class can use a context manager by default and the code simplifies to:

from multiprocessing import Pool

num_threads = 10

with Pool(num_threads) as pool:
    results = pool.map(lst)

Answer 2

使用进程池分配设置数量的进程来执行任务： https ： //docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers