In python, when use multiprocessing Pool, main process can not finished in some situation

Question

first situation, main process can not finished,

from multiprocessing import Pool, Queue

queue = Queue()

def handle(slogan):

    for i in xrange(100000):
        queue.put(slogan)
    print 'put done'  

def main():

    pools = Pool(2)
    for i in xrange(4):
        pools.apply_async(handle, args=('test', ))   
    print 'waiting all done...'

    pools.close()
    pools.join()

    print 'all done...'


if __name__ == '__main__':

    main()

the result of this code, like this:

waiting all done...
put done
put done
put done
put done

I have waited for over 1 hours. I can not understand. I thought multiprocessing module has some bug or something. So I change this code. This time I do not use Queue of multiprocessing, I just use it for computing some numbers. And code as follow:

from multiprocessing import Pool

def handle(slogan):

    tmp = 0
    for i in xrange(100000):
         tmp += i
    print 'put done'


def main():

    pools = Pool(2)
    for i in xrange(4):
        pools.apply_async(handle, args=('test', )) 
    print 'waiting all done...'

    pools.close()
    pools.join()

    print 'all done...'


if __name__ == '__main__':

    main()

for the code, it finished successfully, result as:

waiting all done...
put done
put done
put done
put done
all done...

just because I use Queue? I do not know why. who can explain it for me?

Answer 1

You aren't capturing the result. You should capture the return values from apply_async() and call get() on each one of them.

Also, try specifying a large timeout value in join() or get() . In some versions of Python this is required to work around a bug.

See also: https://stackoverflow.com/a/3571687/4323