使用队列在进程之间进行通信

Question

I have a single process A a the first Pool, and several processes B1..Bk in the second pool and I would like to put items into the queue in A and consume items in B1..Bk.

My first attempt was to just create multiprocessing.Queue and pass it to all those processes. However this gave me the error

RuntimeError: Queue objects should only be shared between processes through inheritance

I found advice that suggests to use multiprocessing.Manager().Queue() instead. But when I do this and try to read from the queue in Bi, I get the error

TypeError: 'AutoProxy[Queue]' object is not iterable

So what is the correct way to do this?

Answer 1

OK, this was just sillyness from my side, I misunderstood what a Queue is! Most importantly a queue is not iterable so one cannot do "for el in somequeue". (My mistake was to think that the queue proxy is not iterable because it is the proxy. However the proxy works fine in place of the actual queue if put/get are used) Also, a (FIFO) queue cannot be closed and does not have a natural "end" which I found annoying because it means one has to send around special "end of queue" entries but not too many of them to not unintentionally block the queue.

So the bottom line is: to share the queue I create a multiprocessing.Manager().Queue() and pass that around, then I use put/get to write/read the queue in different processes and I send some special entry to the reader to indicate end of job.

That a queue cannot get closed and the consumer get an "end of queue" condition is really annoying though, especially when there is an error situation: if a queue is consumed by k consumers, then the writer has to know k and send k end of job indicators and the k consumers all have to be well behaved to retrieve those and shut down. If there is any error, all this cannot be guaranteed any more and eg a consumer may lock or timeout waiting for the end of job indicator that will never arrive.