Celery periodic_task in Django run once(!)

Question

I am creating a periodic_task using celery and Django which I want to run every X seconds.. The task should spawn a couple of sub-tasks, but I need to make sure only one set of sub-tasks are spawned for each main task.

This is what I have..

@periodic_task(run_every=datetime.timedelta(seconds=2))
def initialize_new_jobs():
    for obj in Queue.objects.filter(status__in=['I', 'Q']):
        obj = Queue.objects.get(id=obj.id)
        if obj.status not in ['I', 'Q']:
            continue
        obj.status = 'A'
        obj.save()
        create_other_task.delay(obj.id)

This kinda works, but feels wrong. I haveto refresh obj at the beginning of the loop to make sure another running periodic_task isnt issuing create_other_task on the same Queue object.

Is there any better way of doing this kind of job? Basically, I want to do create_other_task as often as possible, but only ONCE per Queue object with status I or Q.

This is a shortened version of my problem, so please ignore the fact that I could just run create_other_task when creating the Queue object, instead of running the periodic task :)

Answer 1

You could use transactions:

@periodic_task(run_every=datetime.timedelta(seconds=2))
@transaction.commit_on_success
def initialize_new_jobs():
    for obj in Queue.objects.select_for_update().filter(status__in=['I', 'Q']):
        obj.status = 'A'
        obj.save()
        create_other_task.delay(obj.id)

select_for_update() puts an exclusive lock on rows so that other users get blocked when attempting to read the values. The lock is released after the transaction has been committed or rolled back. Reference .

This way you can be sure that the obj has status of I or Q and that the obj.save() will work properly.