How to limit the number of tasks that runs in celery

Question

I have an app running on Heroku and I'm using celery together with a worker dyno to process background work.

I'm running tasks that are using quite a lot of memory. These tasks get started at roughly the same time, but I want only one or two tasks to be running at the same time, the others must wait in the queue. How can I achieve that?

If they run at the same time I run out of memory and the system gets restarted. I know why it's using a lot of memory and not looking to decrease that

Answer 1

Quite simply: limit your concurrency (number of celery worker processes) to the number of tasks that can safely run in parallel on this server.

Note that if you have different tasks having widly different resource needs (ie one task that eats a lot of ram and takes minutes to complete and a couple ones that are fast and don't require much resources at all) you might be better using two distinct nodes to serve them (one for the heavy tasks and the other for the light ones) so heavy tasks don't block light ones. You can use queues to route tasks to different celery nodes.