Python background worker for RabbitMQ

Question

I am building a Python 3 application that will consume messages from RabbitMQ. Is there some Python background job library that can make this easy? I am looking for something similar to Sneakers in Ruby. I would like library to have:

• easy way to define tasks that process RabbitMQ messages (I have a separate non-Python producer application that will create messages and put them into RabbitMQ)
• configure number of worker processes that run tasks
• run workers as daemonized processes

Answer 1

I believe you're looking for Celery

You'll define task as follows

@task
def mytask(param):
    return 1 + 1

It will be put in message broker (for example mentioned RabbitMQ), and then consumed and executed from celery

You can configure number of workers

celery worker --concurrency=10

And yes, it can be demonized

Answer 2

To consume task of RabbitMq you have to define worker, but to run worker in a daemonized mode you have to create a supervisor for that worker

• command to start worker

  • celery worker --concurrency=10 -Ofair --loglevel=DEBUG -A file_name_without_extension -Q queue_name

• steps to create supervisor

  • https://thomassileo.name/blog/2012/08/20/how-to-keep-celery-running-with-supervisor/

  • http://python-rq.org/patterns/supervisor/