Celery + Gevent pool hangs after 1000+ tasks execution

Question

We have 8 Core, 16 GB memory, Linux server running celery, It is running a celery worker queue myQueue , and running with 1000 concurrency under gevent pool.

After executing tasks for around 1 hour, worker suddenly freezes, it is not accepting new tasks from celery beat here is our configuration for celery

App =  Celery('tasks')
class Conf:
   BROKER_URL   = 'amqp://<user>:<pass>@<host>:<port>/<vhost>'
   CELERY_IGNORE_RESULT = True
   CELERY_IMPORTS = ("worker_class",)
   CELERYBEAT_SCHEDULE = {
       'RunTask':{
           'task': 'tasks.worker.MyWorker',
           'schedule' : timedelta(minutes=5)
       }
   }

App.config_from_object(Conf)

we are running celery like below

celery worker --workdir=tasks/ -A worker -P gevent -c 1000 -Q myQueue --loglevel=INFO

And also can someone explain how can I use gevent pool using celery multi

Answer 1

To specify a pool type with celery multi :

celery -A myApp multi start 4 -l INFO -P gevent -c 1000 -Q myQueue

The above command starts 4 gevent workers with each working having a concurrency level of 1000, and all consuming from myQueue .

But that's not where the fun ends, because you can even specify the concurrency of each worker, and also specify which queue each worker consumes. For example:

celery -A myApp multi start 4 -l INFO -P gevent -c:1-3 1000 -c:4 200 -Q:1-2 myQueue1 -Q:3 myQueue2 -Q:4 myQueue3

Like before we start 4 gevent workers, but now workers 1 to 3 have a concurrency of 1000, while the last worker has a concurrency of 200. Also, workers 1 and 2 consume from myQueue1 , worker 3 consumes from myQueue2 , and worker 4 consumes from myQueue4 .

Note: The celery worker options work with celery multi .