Django celery tasks number limit issue

Question

I have problem run celery workers to execute same task multiple times in parallel.

I ran 3 workers and set --concurrency to 2 for all workers.

But it only executes 3 tasks over all 3 workers.

I hope to run about 10 workers.

celery -A my_app worker -l info  -c 2 -n worker1
celery -A my_app worker -l info  -c 2 -n worker2
celery -A my_app worker -l info  -c 2 -n worker3

Please help me whether I can run more than 3 tasks at a time.

Answer 1

I solved it by using max-tasks-per-child argument.

Here is my solution.

celery -A my_app worker -l info -c 10 --max-tasks-per-child 10 -n worker1

After running this, it can execute 10 same tasks at a time in parallel.

Hope this is helpful.