Python Multiprocessing leading to many zombie processes

Question

I have been implementing python's multiprocessing library using a pool of workers. I implemented the following code

import main1
t1 = time.time()
p = Pool(cores) 
result = p.map(main1, client_list[client])
if result == []:
    return []
p.close()
p.join()
print "Time taken in performing request:: ", time.time()-t1
return shorted(result)

However, after running the process for a while, I get lot of running background processes of my app. Here is a snapshot after doing ps aux for my app

Now, I have read a lot of similar questions on stackoverflow like how to kill zombie processes created by multiprocessing module? which calls for using .join() which I have already implemented and I learned how to kill all these processes from here Python Multiprocessing Kill Processes . But I want to know what possibly could go wrong with my code. I won't able to share all of my code in the main1 function but I have put the entire code block in try catch block to avoid cases where an error in the main code could lead to zombie processes.

def main1((param1, param2, param3)):
    try:
       resout.append(some_data) //resout in case of no error
    except:
        print traceback.format_exc()
        resout = []  //sending empty resout in case of error
    return resout

I'm still very new to the concept of parallel programming and debugging issues with it is turning out to be tricky.Any help will be greatly appreciated.

Answer 1

Usually the most common problem is that the pool is created but it is not closed.

The best way I know to guarantee that the pool is closed is to use a try/finally clause:

try:
    pool = Pool(ncores)
    pool.map(yourfunction, arguments)
finally:
    pool.close()
    pool.join()

If you don't want to struggle with multiprocessing , I wrote a simple package named parmap that wraps multiprocessing to make my life (and potentially yours) easier.

pip install parmap

import parmap
parmap.map(yourfunction, arguments)

From the parmap usage section:

• Simple parallel example:

   import parmap y1 = [myfunction(x, argument1, argument2) for x in mylist] y2 = parmap.map(myfunction, mylist, argument1, argument2) y1 == y2 

• Iterating over a list of tuples:

   # You want to do: z = [myfunction(x, y, argument1, argument2) for (x,y) in mylist] z = parmap.starmap(myfunction, mylist, argument1, argument2) # You want to do: listx = [1, 2, 3, 4, 5, 6] listy = [2, 3, 4, 5, 6, 7] param = 3.14 param2 = 42 listz = [] for (x, y) in zip(listx, listy): listz.append(myfunction(x, y, param1, param2)) # In parallel: listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)