Python Multiprocessing queue

Question

I am populating a queue with a set of jobs that I want to run in parallel and using python's multiprocessing module for doing that. Code snippet below:

import multiprocessing
from multiprocessing import Queue
queue = Queue()
jobs = [['a', 'b'], ['c', 'd']]
for job in jobs:
    queue.put(job)

When I do queue.get() I get the following:

['a', 'b']

Why is the queue not getting populated with all the jobs?

Answer 1

The queue is actually geting populated. You need to call queue.get() for each time you put an object to the queue. So you just need to call queue.get() one more time.

>>> import multiprocessing
>>> from multiprocessing import Queue
>>> queue = Queue()
>>> jobs = [['a', 'b'], ['c', 'd']]
>>> for job in jobs:
    queue.put(job)


>>> queue.get()
['a', 'b']
>>> queue.get()
['c', 'd']

Answer 2

The queue is getting populated with all your jobs. queue.get() will

Remove and return an item from the queue.

An item is singular. If you want to drain the queue, then just put your .get() in a loop, but be sure to catch the Empty exception.