Celery + RabbitMQ Results Backend + WebSockets?

Question

I have a Flask web app and there are some tasks that are long running (> 1 minute). I would like to offload these from flask (blocking web server, etc) and move them into external tasks. I got Celery setup and am able to run these tasks from Flask. Good.

To notify users when the task is done and return the results, I wanted to use web sockets. This is where it gets hard.

I have a gevent websocket server running that my client connects to. When I launch the celery task I pass in the session ID of the client so that when I get my results I know who they belong to.

Now I need the websocket server to know when these tasks are done, look at the results, and send them to the appropriate connection.

How? This is where I get completely lost. I can listen for celery events with celery.events.EventListener.capture , but this blocks my websocket server so I can't have multiple connections!

How can I listen for celery events without blocking?

One idea is to make the request over the web socket and then launch the celery task from my websocket server. That would solve the notification issue (just used celery events), but I'm still curious about my original question where the request was made to my standard flask server.

Answer 1

You need your flask app not to block on the capture call. One solution is to run flask with gevent or eventlet so that your process will handle request asynchronously and blocking calls will not block the process.

There are tons of examples about this, to name one: https://gist.github.com/lrvick/1185629