Python: Is there a way to schedule Celery tasks to run only within a specific time frame?

Question

I have an app that uses Celery and I would like it to only allow the workers to run the tasks between certain times of the day.

For example:

• I'd want my tasks to only be executed between 2:00 and 4:00 everyday
• If all tasks aren't completed from the queue in that time frame, then they are carried over to the next day to be run between 2:00 and 4:00
• I can add more tasks to the queue at any time

I've been looking at Celery's Periodic Tasks but this only seems to be able to set the time in which tasks are exectuted without setting a time at which they should cease:

from celery.schedules import crontab

CELERYBEAT_SCHEDULE = {
    # Executes every Monday morning at 7:30 A.M
    'add-every-monday-morning': {
        'task': 'tasks.add',
        'schedule': crontab(hour=7, minute=30, day_of_week=1),
        'args': (16, 16),
    },
}

There might not be a way to do this with Celery out of the box but I just wanted to check before I started to build something that could automate this.

Cheers in advance!

Answer 1

I think the best you can do with what's available natively in celery is set the expires option on your task to 2 hours.

CELERYBEAT_SCHEDULE = {
    # Executes every Monday morning at 7:30 A.M
    'add-every-monday-morning': {
        'task': 'tasks.add',
        'schedule': crontab(hour=14),
        'expires': 60 * 60 * 2,  # 2 hours
    ...

The rest of the functionality you will need to implement yourself.

Answer 2

You can dynamically add and cancel consumers for queues . Just set up a cronjob that adds the consumers at 2am, remove them at 4am. You could also use a scheduled celery task, but then you have to set up a second queue to ensure it gets executed.

Just make sure that your queue is able to hold all tasks that are enqueued in the meantime.