How to run periodic tasks concurrently with Celery and Django

Question

I have some tasks being running by celery in my Django project.

I use contrab to specify the time the task should be run, like this:

from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    'task_a': {
        'task': 'tasks.task_a',
        'schedule': crontab(minute=0, hour='5,18'),
    },
    'task_b': {
        'task': 'tasks.task_b',
        'schedule': crontab(minute=4, hour='5,18'),
    },
}

What has been happening is that one task is executed, and only about 5 minutes later the other starts. When they should be executed at the same time.

I would like all of them to be started at the same, but this it's not what is happening

there are about eight tasks in total, some of which take a long time to complete

I am using the following command at the moment

initially, it was like this

celery -A api worker  --concurrency=4 -n <name>

then I tried

celery -A api multi  --concurrency=4 -n <name>

and finally

celery -A api multi -P gevent --concurrency=4 -n <name>

They are all shared_tasks

@shared_task(bind=True, name="tasks.task_a")
def task_a(self):
     pass

and I'm using autodiscover_tasks

app = Celery('<app-name>')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Answer 1

Sounds like it might be a hardware issue. Make sure you have enough CPU resource available to the celery worker.

Concurrency should be set to the number of CPUs available on your machine (which is default). If your tasks are CPU bound, the pool should be set to prefork (which is default) and not gevent (which is for I/O bound like HTTP requests)

Celery multi is to start multiple works. Generally Celery's default advise is 1 worker per machine. But they do tell you to do your own testing.

Edit: This link does a great job at explaining: https://www.distributedpython.com/2018/10/26/celery-execution-pool/