I have 3 remote workers, each one is running with default pool (prefork) and single task.
A single task is taking 2 to 5 minutes for completion as it runs on many different tools and inserts database in ELK.
worker command: celery -A project worker -l info
Which pool class should I use to make processing faster?
is there any other method to improve performance?
funny that this question scrolled by.
We just switched from eventlet to gevent. Eventlet caused hanging broker connections which ultimately stalled the workers.
General tips:
More advanced options to tune your celery workers:
I use celery in my django project.
When use eventlet I got a lot of errors like
dns timeout
in eventlet==0.25.2
Another error:
GreenSSLSocket does not have a public constructor
in eventlet==0.22.0
. So Sad.
So I replace eventlet with gevent==20.6.2
, and Everything works perfectly
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.