使用multiprocessing.Pool时如何理解multiprocessing.Queue？

Question

Why can't I put process in Pool into a Queue ?
Here my code works when using Pool and can get Test instance attributes.

from multiprocessing import Pool
from multiprocessing import Queue


class Test(object):
    def __init__(self, num):
        self.num = num


if __name__ == '__main__':
    p = Pool()
    procs = []
    for i in range(5):
        proc = p.apply_async(Test, args=(i,))
        procs.append(proc)
    p.close()
    for each in procs:
        test = each.get(10)
        print(test.num)
    p.join()

When I try to use Queue not python list to store processes, this won't work.

My code:

from multiprocessing import Pool
from multiprocessing import Queue


class Test(object):
    def __init__(self, num):
        self.num = num


if __name__ == '__main__':
    p = Pool()
    q = Queue()
    for i in range(5):
        proc = p.apply_async(Test, args=(i,))
        q.put(proc)
    p.close()
    while not q.empty():
        q.get()
    p.join()

Error msg:

Traceback (most recent call last):
  File "C:\Users\laich\AppData\Local\Programs\Python\Python36- 
32\lib\multiprocessing\queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "C:\Users\laich\AppData\Local\Programs\Python\Python36- 
32\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

I go see the multiprocessing doc:

class multiprocessing.Queue([maxsize]) Returns a process shared queue implemented using a pipe and a few locks/semaphores. When a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe.

The usual queue.Empty and queue.Full exceptions from the standard library's queue module are raised to signal timeouts.

Queue implements all the methods of queue.Queue except for task_done() and join() .

Here it says "puts an item", this item can't be anything (python object)? In my case I try to put process in Pool() into Queue .

Answer 1

There are at least two problems with your Queue -based code. Pool.apply_async method returns an AsyncResult object, not a process. You can call get on this object to obtain the result of the corresponding process. With this difference in mind let's look at your code:

proc = p.apply_async(Test, args=(i,)) # Returns an AsyncResult object
q.put(proc) # won't work

The second line will always fail in your case. Anything that you put in a queue must be picklable, because multiprocess.Queue uses serialization. This is not well documented and there is an open issue in Python's issue tracker to update the documentation. The problem is that AsyncResult is not picklable. You can try yourself:

import pickle
import multiprocessing as mp

with mp.Pool() as p:
    result = p.apply_async(lambda x: x, (1,))

pickle.dumps(result) # Error

AsyncResult contains some lock objects internally and they are not serializable. Let' move to the next problem:

while not q.empty():
    q.get()

If I'm not wrong, in the code above you want to call AsyncResult.get and not Queue.get . In this case you have to first get your object from the queue and then call the corresponding method on your object. However this is not the case in your code, since AsyncResult is not serializable.

Answer 2

As @Mehdi Sadeghi explained , AsyncResult objects can't be pickled, which multiprocessing.Queue s requires. However you don't need one here because the queue isn't being shared among the processes. This mean you can just use a regular Queue .

from multiprocessing import Pool
#from multiprocessing import Queue
from queue import Queue


class Test(object):
    def __init__(self, num):
        self.num = num
        print('Test({!r}) created'.format(num))


if __name__ == '__main__':
    p = Pool()
    q = Queue()
    for i in range(5):
        proc = p.apply_async(Test, args=(i,))
        q.put(proc)
    p.close()
    while not q.empty():
        q.get()
    p.join()

    print('done')

Output:

Test(0)
Test(1)
Test(2)
Test(3)
Test(4)
done