multiprocessing creates zombie process even after .join()

Question

I have a chunk of code that I am parallelizing in the following manner:-

def stats_wrapper(a,b,c,d):
    q = mp.Queue()
    # b here is a dictionary
    processes = [mp.Process(target = stats,args = (a,{b1:b[b1]},c,d,q) ) for b1 in b]
    for p in processes:
        p.start()
    results = []
    for p in processes:
        results.append(q.get())
    return(results)

I am seeing a lot of zombie processes after I execute this block. I am trying to use the .join() method the following way before the return(result) line :-

for p in processes:
   p.join()

but that doesn't help with getting rid of the zombie processes. Can someone help me identify exactly where am I going wrong with my code?

Edit: - I am using elsewhere in my code another approach to parallelization which is again giving me a number of zombie processes,except I don't know how to refactor the code to add a join.

q = mp.Queue()
jobs = (func1,func2,func3)
args = ((arg1,arg2,arg3),(arg2,arg3),(arg1,arg4))
for job,arg in zip(jobs,args):
   mp.Process(target = job,args = arg,name = str(job.__name__)).start()
result = []
for _ in range(len(job)): 
   result.append(q.get())

Answer 1

If you're willing to try the higher-level multiprocessing.Pool() ,

def stats_wrapper(a, b, c, d):
    with multiprocessing.Pool() as p:
        args = [(a, {b1: b[b1]}, c, d) for b1 in b]
        return list(p.starmap(stats, args))

should be equivalent (aside from q not being passed to stats ; that function can simply return the result).

If you don't need the results to be in order, use p.imap_unordered() , which can use the process pool more efficiently.

Answer 2

You can try this code for getting results out of queue:

def stats_wrapper(a,b,c,d):
    q = mp.Queue()
    # b here is a dictionary
    processes = [mp.Process(target = stats,args = (a,{b1:b[b1]},c,d,q)) for b1 in b]
    for p in processes:
        p.start()
    results = []
    for p in processes:
        p.join()
    while not q.empty():
        results.append(q.get())
    return(results)