Python 3.8 websocket echo client with queue: asyncio.Queue get() does not get queue items added on the fly

Question

I try to create a client which uses a asyncio.Queue to feed the messages I want to send to the server. Receiving data from websocket server works great. Sending data which is just generated by the producer works, too. For explaning what works and what fails, first here's my code:

import sys
import asyncio
import websockets


class WebSocketClient:
    def __init__(self):
        self.send_queue = asyncio.Queue()
        #self.send_queue.put_nowait('test-message-1')

    async def startup(self):
        await self.connect_websocket()
        consumer_task = asyncio.create_task(
            self.consumer_handler()
        )
        producer_task = asyncio.create_task(
            self.producer_handler()
        )
        done, pending = await asyncio.wait(
            [consumer_task, producer_task],
            return_when=asyncio.ALL_COMPLETED
        )
        for task in pending:
            task.cancel()

    async def connect_websocket(self):
        try:
            self.connection = await websockets.client.connect('ws://my-server')
        except ConnectionRefusedError:
            sys.exit('error: cannot connect to backend')

    async def consumer_handler(self):
        async for message in self.connection:
            await self.consumer(message)

    async def consumer(self, message):
        self.send_queue.put_nowait(message)
        # await self.send_queue.put(message)
        print('mirrored message %s now in queue, queue size is %s' % (message, self.send_queue.qsize()))

    async def producer_handler(self):
        while True:
            message = await self.producer()
            await self.connection.send(message)

    async def producer(self):
        result = await self.send_queue.get()
        self.send_queue.task_done()
        #await asyncio.sleep(10)
        #result = 'test-message-2'
        return result


if __name__ == '__main__':
    wsc = WebSocketClient()
    asyncio.run(wsc.startup())

Connecting works great. If I send something from my server to the client, this works great too and prints the message in consumer(). But producer never gets any message I put in send_queue inside consumer() .

The reason why I chose send_queue.put_nowait in consumer() was that I wanted to prevent deadlocks. If I use the line await self.send_queue.put(message) line instead of self.send_queue.put_nowait(message) it makes no difference.

I thought, maybe the queue dos not work at all, so I filled something to the queue just at creation in __init__() : self.send_queue.put_nowait("test-message-1") . This works and is sent to my server. So the basic concept of the queue and await queue.get() works.

I als thought, maybe there is some issue with the producer, so let's just randomly generate messages during runtime: result = "test-message-2" instead of result = await self.send_queue.get() . This works too: every 10 seconds 'test-message-2' is sent to my server.

EDIT: This also happens if I try to add stuff from another source to the queue on the fly. I build a small asyncio socket server which pushes any message to the queue, which works great, and you can see the messages I added from the other source with qsize() in consumer() , but still no successfull queue.get() . So the queue itself seems to work, just not get() . This is btw the reason for the queue, too: I would like to send data from quite different sources.

So, this is the point where I'm stuck. My wild guess is that the queue I use in producer() is not the same as in consumer() , something which happens at threading quite easily if you use non-thread-safe queues like asyncio.Queue, but as I understood it I don't use threading at all, just coroutines. So, what else went wrong here?

Just for the context: it's a Ubuntu 20.04 python 3.8.2 inside a docker container.

Thanks, Ernesto

Answer 1

Just for the records - the solution for my problem was quite simple: I defined send_queue outside the event loop created by my websocket client. So it called events.get_event_loop() and got its own loop - which was not part of the main loop and therefore never called, therefore await queue.get() really never got anything back.

In normal mode, you don't see any message which is a hint to this issue. But, python documentation to the rescue: for course they mentioned it at https://docs.python.org/3/library/asyncio-dev.html : logging.DEBUG gave the hints I needed to find the problem.

Answer 2

It should look like this:

class WebSocketClient:
    async def startup(self):
        self.send_queue = asyncio.Queue()
        await self.connect_websocket()

Then the queue is defined inside the main loop.