Python multiprocessing queue manager address already in use

Question

Currently I'm trying to implement communication between different processes in Python 3.6.7. My idea was to use queues, because named pipes were not sufficient.

I have a listener implemented as follows:

result_queue = queue.Queue()
BaseManager.register('queue', callable=lamda: result_queue)
queue_manager = BaseManager(address=('127.0.0.1', 50000))
queue_manager.start()
while do_run:
   data = result_queue.get()
   print(data)
queue_manager.shutdown()

And I have a writer defined as follows:

BaseManager.register('queue')
manager = BaseManager(address=('127.0.0.1', 50000))
manager.connect()
queue = manager.queue()
queue.put(message)

The issue is that when I'm running my unit tests in a Docker environment I keep getting the following error:

 OSError: [Errno 98] Address already in use

It seems to run fine for the first test but keeps on having the error above for all the tests that follow.

My guess is that the BaseManager comes across a 'TIME_WAIT' on the port but it seems that I cannot set an option to re-use the socket instead of waiting.

Am I doing something wrong here or are there ways to force the re-use of the sockets?

Answer 1

An easy way to fix this sort of thing is to let the OS choose a port:

queue_manager = BaseManager(address=('127.0.0.1', 0)) # setting the port to 0 allows the OS to choose.
print(queue_manager.address)

Since you are starting the listener and the writer from a common parent process, you can do the above, capture the address somewhere (pipe, file, whatever), then pass the port number to the other process.

This way, you can even run multiple instances of your test in parallel on the same machine at the same time without conflict. The OS will make sure you get a port which is not in use each time.