How to iterate through a Python Queue.Queue with a for loop instead of a while loop?

Question

Normally we code it like this:

while True:
    job = queue.get()
    ...

Answer 1

You can use iter with callable. (You should pass two arguments, one for the callable, the other for the sentinel value)

for job in iter(queue.get, None): # Replace `None` as you need.
    # do stuff with job

NOTE This will block when no elements remain and no sentinel value is put. Also, like a while - get loop and unlike normal for loops over containers, it will remove items from the queue.

UPDATE : None is common value, so here's a sample with more concrete sentinel value:

sentinel = object()
for job in iter(queue.get, sentinel):
    # do stuff with job

Answer 2

I would say this is an easy way to iterate over queue in some points:

from queue import Queue

q = Queue()
q.put(1)
q.put(2)
q.put(3)

for i in q.queue:
    print(i)

Answer 3

For that kind of queue actually I would not typically use this check of queue.empty() because I always use it in a threaded context and thus cannot know whether another thread would put something in there in a few milliseconds (thus that check would be useless anyway). I never check a queue for being empty. I rather use a sentinel value which marks the ending of a producer.

So using the iter(queue.get, Sentinel) is more what I like.

If you know that no other thread will put items in the queue anymore and just want to drain it from all currently contained items, then you can use sth like this:

class Drainer(object):
  def __init__(self, q):
    self.q = q
  def __iter__(self):
    while True:
      try:
        yield self.q.get_nowait()
      except queue.Empty:  # on python 2 use Queue.Empty
        break

for item in Drainer(q):
  print(item)

or

def drain(q):
  while True:
    try:
      yield q.get_nowait()
    except queue.Empty:  # on python 2 use Queue.Empty
      break

for item in drain(q):
  print(item)

Answer 4

My first though was for the iter function, but the built in queue module doesn't return a sentinel, so a good alternative might be to define your own wrapper class:

import Queue

class IterableQueue():
    def __init__(self,source_queue):
            self.source_queue = source_queue
    def __iter__(self):
        while True:
            try:
               yield self.source_queue.get_nowait()
            except Queue.Empty:
               return

This iterator wraps the queue and yields until the queue is empty, then returns, so now you can do:

q = Queue.Queue()
q.put(1)
q.put(2)
q.put(3)

for n in IterableQueue(q):
    print(n)

Output:

1
2
3

This method is a bit verbose it would be interesting if anyone knows anything better using the builtins.

Answer 5

I would like to mention that I have seen situations where I have seen that the loop:

for job in iter(queue.get, None):

behaved differently than

while True:
  data = queue.get()
  if data is None:
    break

if there were multiple processes in play. I don't have a reproducible example, but the former simply ended earlier than the latter. I think it may be something with ordering and stuff, but dunno, really...

If process A was filling a queue by reading rows from file and transforming them. And the process B was consuming from this queue, it happened to me that with the first implementation, the process B ended after eg 6 items even though I know that the process A processed 10 items and then put sentinel None in the queue.

Answer 6

I guess this is the easier way

for elem in list(q.queue):
      print(elem)