How to pull from a different queue inside Celery task

Question

I have a task queue, queue A, and a resource queue, queue B. At program start, there are a finite number of resources put in queue B. Tasks may execute if they're able to pull a resource from queue B, and once they're done executing they must put the resource back in queue B.

Using Celery, is there a way to pull from a different queue inside a task to accomplish this? Or does this need to be architected completely differently?

An alternative, albeit with potentially reduced concurrency, is pairing the task with a resource before it's put in the task queue, then polling the task for completion, and on completion freeing up that resource and putting it back in the task queue with a different task. Is this the standard solution?

Answer 1

The architecture I ended up deciding on accomplishes this with the help of the chain primitive in Celery, with no loss in concurrency.

To ensure that no two tasks are using the same resource concurrently, you can divide all tasks among the available resources before adding the tasks to the queue (so say you had 10 tasks, and 3 resources, simply assign 3 tasks to two of the resources, and 4 to the third resource). Then, for each resource, call the tasks with chain() which tells Celery to execute each of the tasks for that resource in series.