Why are django-q scheduled tasks delayed randomly?

Question

I'm finding that django-q is not executing tasks I schedule on time. There can be delays of several seconds to almost a minute.

I schedule a task like so:

from django.utils import timezone
from django_q.models import Schedule

def schedule_auction_deadlines(self):
    now = timezone.now()
    if self.deadline:
        name = "end_phase_%d" % self.id
        Schedule.objects.filter(name=name).delete()
        if now < self.deadline:
            Schedule.objects.create(name=name, func="myapp.views.end_phase", args=str(self.id), next_run=self.deadline, schedule_type=Schedule.ONCE)

And here is my configuration in the settings.py file:

Q_CLUSTER = {
    'name': 'myproj',
    'label': 'Django Q',
    'timeout': 30,
    'catch_up': True,
    'guard_cycle': 1,
    'redis': os.environ.get('REDIS_URL', 'redis://localhost:6379'),
}

From the docs it seemed like guard_cycle might be relevant, but I've already got it set at the minimum setting.

What could be causing these delays?

Answer 1

I'm finding that django-q is not executing tasks I schedule on time. There can be delays of several seconds to almost a minute.

According to the docs , the scheduler is only checked every thirty seconds, so it's not surprising if it takes 30-40 seconds.

From the docs it seemed like guard_cycle might be relevant, but I've already got it set at the minimum setting.

Guard cycle actually defaults to 0.5, so you have it set higher than the default. But the scheduler interval is implemented a little strangely, because setting a lower guard time can actually increase the amount of time between scheduler checks.

The important code is in Sentinel.guard :

def guard(self):
    logger.info(
        _(
            f"{current_process().name} guarding cluster {humanize(self.cluster_id.hex)}"
        )
    )
    self.start_event.set()
    Stat(self).save()
    logger.info(_(f"Q Cluster {humanize(self.cluster_id.hex)} running."))
    counter = 0
    cycle = Conf.GUARD_CYCLE  # guard loop sleep in seconds
    # Guard loop. Runs at least once
    while not self.stop_event.is_set() or not counter:
        # Check Workers
        [...]
        # Check Monitor
        [...]
        # Check Pusher
        [...]
        # Call scheduler once a minute (or so)
        counter += cycle
        if counter >= 30 and Conf.SCHEDULER:
            counter = 0
            scheduler(broker=self.broker)
        # Save current status
        Stat(self).save()
        sleep(cycle)
    self.stop()

So the scheduler is not called every time through the loop. It is only called every time counter is greater than 30, and counter counts up by guard_cycle each time. If you set guard_cycle to 0.001, the guard would only call the scheduler once every 30,000 times through the loop, which would take much more than 30 seconds, due to all the other work the guard does in checking the health of the cluster.

Since the scheduler interval is hard-coded, I don't see what you could do besides modify Django Q or call the scheduler yourself. The docs don't say anything about how to do this.