How do I override the backend for celery tasks

Question

we're using Redis as our result backend. However for one task we'd like to override this to use RabbitMQ instead.

The documentation for Task.backend says:

The result store backend to use for this task. Defaults to the CELERY_RESULT_BACKEND setting

So I had assumed that we could set Task.backend to a string of the same format accepted by CELERY_RESULT_BACKEND .

So I try this:

celeryconfig.py

CELERY_RESULT_BACKEND = "redis://redis-host:7777"

tasks.py

@app.task(backend='amqp://guest@localhost/tasks-stg')
def my_task(params):
    ...

However the worker fails with :

[2015-05-07 13:33:49,264: ERROR/Worker-1] Process Worker-1
Traceback (most recent call last):
  File "/project/python2.7_x64/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
    self.run()
  File "/project/python2.7_x64/lib/python2.7/site-packages/billiard/pool.py", line 286, in run
    self.after_fork()
  File "/project/python2.7_x64/lib/python2.7/site-packages/billiard/pool.py", line 389, in after_fork
    self.initializer(*self.initargs)
  File "/project/python2.7_x64/lib/python2.7/site-packages/celery/concurrency/prefork.py", line 81, in process_initializer
    app=app)
  File "/project/python2.7_x64/lib/python2.7/site-packages/celery/app/trace.py", line 178, in build_tracer
    store_result = backend.store_result
AttributeError: 'str' object has no attribute 'store_result'

Answer 1

The documentation is incorrect. Task.backend is actually an instance of a backend class from celery.backends . In this case to override the task class I had to do this:

from celery.backends.amqp import AMQPBackend

@app.task(backend=AMQPBackend(app, url='amqp://guest@localhost/tasks-stg'))
def my_task(params):
    ...

However the workers continue to use the default class and don't seem to offer a way to override this.

Answer 2

Extend & update the original accepted answer, those who hit similar issue might be interested with List of Options available in Celery task decorator . See Task.backend section :

The result store backend to use for this task. An instance of one of the backend classes in celery.backends . Defaults to app.backend

Also, AMQP result backend has been removed since the version 5.0 , instead you could use celery.backends.rpc.RPCBackend .

For example :

from celery.backends.rpc import RPCBackend as CeleryRpcBackend
_rpc_backend = CeleryRpcBackend(app=your_celery_app, \
        exchange=RPC_REPLY_EXCHANGE_DEFAULT_NAME, \
        exchange_type=RPC_EXCHANGE_DEFAULT_TYPE )

@your_celery_app.task(backend=_rpc_backend, ... OTHER_ARGS ... )
def your_task_function(**kwargs):
    DO_SOMETHING ...