Scale Celery app on ACI based on tasks in queue

Question

We need celery to scale in order to avoid long queues on limited celery workers and instead spawn new celery containers in ACI to distribute those tasks.

In order to achieve this scaling (spawning of new celery containers in ACI), we need to know the number of tasks that are waiting to be executed and accordingly spawn those many containers on ACI.

Any suggestions would be greatly appreciated.

Answer 1

Celery used message brokers to store tasks and supports multiple message brokers like rabbitmq, sqs etc. You can monitor the size of the queue at the message broker which can provide you information about pending tasks and perform the scale up of the celery workers.

In case of rabbitmq, You can read about queue metrics from link and perform action basis the number of pending messages.