mpi4py - deadlock with scatter & gather loop

Question

Hi I am using mpi4py to play around with MPI. My usecase is that I have a Python Queue object which holds tasks to be processed like such:

from queue import Queue
my_queue = Queue()
my_queue.put({'task': [1, 2, 3]})
# while True:
if comm.rank == 0:
  task = my_queue.get()
else:
  task = None
work = comm.scatter(task, root=0)
calc = do_calculation(work) # whatever calculation
result = comm.gather(calc, root=0)
if comm.rank == 0:
  print(result)

This is working just fine and if I keep appending tasks (so in the beginning I put 2 tasks into the queue and just copy the upper piece of code) it works as well. My goal is now to let this piece of code run in an infinite loop and whenever something is put into the queue (with ie a seperate thread etc.) it should be processed (my_queue.get() should be blocking so that's not the problem).

But when I try to wrap this piece of code in an infinite loop while True: (see comment) the program does not produce any output and just locks up (seems like a deadlock).

Answer 1

I have found the answer to my issue. There was no deadlock. The code functioned perfectly the only problem was that the buffer of stdout was not flushed. This resulted that the results were not printed to the console and I assumed that the code produced a deadlock.

Adding the line sys.stdout.flush() to manually flush the buffer prints the results to the console and resolved my issue.