使用两个队列同步线程

Question

Assume that I have a pool of threads. This pool of threads uses two queues q1 and q2 . It reads from q1 and writes new items on q2 . When q1 is empty, we swap the two queues q1, q2, = q2, q1 and we repeat the process until both are empty. In order to synchonize the threads, I use another queue which contains only one item and I remove this item at the end of the process.

I think this is a very stupid way to do it. Any suggestion for improvement?

A very nice simple example is on the docs but with only one queue. My solution does not look very nice, if it is correct:

global flag
global lock
global barrier
global q1
global q2
global q
while True:
    if q1.empty():
        flag = False
        barrier.wait() # wait for all the theads to reach this point.
        # execute the code of swapping queues only once
        with lock:
            if not flag:
                flag = True
                if q2.empty():
                    q.get()
                    q.task_done()
                else:
                    q1, q2 = q2, q1

    process_items_in_q1()

Answer 1

I think your aproach works. Here are two others that you may or may not find better:

1. you can use only one queue and push N markers to separate the "levels", where N is the number of threads. When a thread reads a marker it just calls barrier.wait(). This should be enough if the goal is just to ensure that all N threads call barrier.wait() between the levels.

2. or alternatively, you can simplify a bit the code posted above: use 2 plain lists instead of 2 queues. All threads pop from list1 and append to list2 , which doesn't need the special care in the logic of Queue. Also, if the "levels" are each relatively large, you might simply repeat for each level the whole code that creates threads and waits until they are all done. This would make the one-level-at-a-time logic even more obvious.

(Finally, as usual, note that so far you don't gain any processing power by using threads in Python: there's the GIL.)