How do I move tasks from one queue to another in celery

Question

I need to be able to do two things, both of which depend on my being able to move tasks from one queue to another.:

1. when task fails certain number of retries, move it to another queue
2. when task meets certain conditions, move it to different queue (where it will be processed by a different worker)

Do note that I mean to put stuff in another queue from one task in a worker - not from the main app.

The only piece of code that I can find to do this is referenced here - https://stackoverflow.com/a/27144119/112050 can someone point out the correct api to do this ?

Answer 1

Your idea of moving a task actually comes down to running the same task with the same parameters but sending it to a different queue.

apply_async has a queue parameter

from celery.exceptions import MaxRetriesExceededError

@shared_task(default_retry_delay = 1 * 60, max_retries = 10)
def some_task(arg1, arg2):
    try:
        # task logic

        if some_condition:
            some_task.apply_async([arg1, arg2], queue='different_queue')
            return

    except MaxRetriesExceededError:
        some_task.apply_async([arg1, arg2], queue='different_queue')

    except Exception, exc:
        raise some_task.retry(exc=exc) 

If you actually want to move tasks between queues, assuming you are using RabbitMQ, you can use the Shovel Plugin . For example to move messages from queue q1 to queue q2 on the local broker:

rabbitmqctl set_parameter shovel my-shovel \
'{"src-uri": "amqp://", "src-queue": "q1", \
 "dest-uri": "amqp://", "dest-queue": "q2"}'