Python's multiprocessing.Pool spawns new processes

Question

I have a simple question about the multiprocessing module. I am using multiprocessing.Pool's map() function to speed up execution of self-written code on my local machine. However, this code is run in an iterative loop and I find additional Python processes spawned in my machine with every iteration of the loop. (This is a problem because the system slowly grinds to a halt). Here's a simple example:

from multiprocessing import Pool
import os

nthreads = 2
for ii in xrange(5):
    pool = Pool(processes=nthreads)  # (in my code, Pool is inside a pickleable function.)
    runningProcesses = os.popen('ps | grep ython').readlines()
    nproc = len(runningProcesses)
    print "After iteration %i there were %i Python processes running!" % (ii, nproc)

The output is:

After iteration 0 there were 5 Python processes running!
After iteration 1 there were 7 Python processes running!
After iteration 2 there were 9 Python processes running!
After iteration 3 there were 11 Python processes running!
After iteration 4 there were 13 Python processes running!

How should I arrange my code to avoid spawning many new Python processes? I am running Python 2.7.6, which has multiprocessing v0.70a1, and am on a 4-core MacBook Pro running OSX 10.8.5.

Answer 1

将pool = Pool(processes=nthreads)放在for循环上方

Answer 2

As discussed in the comments - the worker processes in the pool are not being closed/joined, so they never terminate. The top answer here shows how to clean up the pool when you no longer need it: Python multiprocessing pool, join; not waiting to go on?

As a side note, if you are creating large numbers of workers and using them to perform very short/quick jobs, then you may find that the performance suffers - there is an overhead for the OS to create and destroy processes. If that is the case, then you should look at using a single Pool throughout your application.