How does this loop works: for item in iter(queue.get, None)?

Question

I'm using python Queue to store items to be processed by threads. From what I've read online, putting a 'None' object in the queue and setting up the thread processing like this will make the thread stop. (which it does)

for item in iter(queue.get, None):
    #do stuff
    queue.task_done()

Now I cannot find many information online about this type of for loop. From what I've seen it just ends and will not process anything else which leaves the None object in the queue. Adding a queue.task_done() at the end doesn't work.

Can someone explain to me more about this type of for loops. How are they named and how do they work in general, or point me towards some good documentation about it since I cannot find any.

Answer 1

This line:

for item in iter(queue.get, None):

Reads

"Keep calling queue.get , and then run the loop body, until queue.get() == None

Answer 2

See the documentation for the iter() function :

If the second argument, sentinel , is given, then o must be a callable object. The iterator created in this case will call o with no arguments for each call to its next() method; if the value returned is equal to sentinel , StopIteration will be raised, otherwise the value will be returned.

So the loop's behavior is equivalent to:

sentinel = None
while True:
  item = queue.get()
  if item == sentinel:
    break
  # do stuff
  queue.task_done()

Answer 3

It is a second form of iter() :

 iter(callable, sentinel) -> iterator 

In the second form, the callable is called until it returns the sentinel.

(from help(iter) ).

So each time for-statement tries to get an element from iterator returned by iter , it calls queue.get() , checks if returned not None, and passed returned value as item to for-statement-body. When somebody pushes None-terminator onto queue, queue.get() returns None causing iterator to throw StopIteration and interpreter leaves for-statement block.

Answer 4

I think the curial part is that:

NOTE This will block when there's no element remain and no sentinel value is put.

so basically this method does not automatically run on any queue, you have to put a None value to indicate the end of process

Answer 5

I noticed that the approaches differ in a specific multiprocessing environment. Described here