带有芹菜的 Django - 找不到现有对象

Question

I am having problem with executing celery task from another celery task.

Here is the problematic snippet (data object already exists in database, its attributes are just updated inside finalize_data function):

def finalize_data(data):
    data = update_statistics(data)
    data.save()
    from apps.datas.tasks import optimize_data
    optimize_data.delay(data.pk)

@shared_task
def optimize_data(data_pk):
    data = Data.objects.get(pk=data_pk)
    #Do something with data

Get call in optimize_data function fails with "Data matching query does not exist."

If I call the retrieve by pk function in finalize_data function it works fine. It also works fine if I delay the celery task call for some time.

This line:

optimize_data.apply_async((data.pk,), countdown=10)

instead of

optimize_data.delay(data.pk)

works fine. But I don't want to use hacks in my code. Is it possible that .save() call is asynchronously blocking access to that row/object?

Answer 1

I know that this is an old post but I stumbled on this problem today. Lee's answer pointed me to the correct direction but I think a better solution exists today.

Using the on_commit handler provided by Django this problem can be solved without a hackish way of countdowns in the code which might not be intuitive to the user about why it exsits.

I'm not sure if this existed when the question was posted but I'm just posting the answer so that people who come here in the future know about the alternative.

Answer 2

I'm guessing your caller is inside a transaction that hasn't committed before celery starts to process the task. Hence celery can't find the record. That is why adding a countdown makes it work.

A 1 second countdown will probably work as well as the 10 second one in your example. I've used 1 second countdowns throughout code to deal with this issue.

Another solution is to stop using transactions.

Answer 3

You could use an on_commit hook to make sure the celery task isn't triggered until after the transaction commits?

DjangoDocs#performing-actions-after-commit

It's a feature that was added in Django 1.9.

from django.db import transaction

def do_something():
    pass  # send a mail, invalidate a cache, fire off a Celery task, etc.

transaction.on_commit(do_something)

You can also wrap your function in a lambda:

transaction.on_commit(lambda: some_celery_task.delay('arg1'))

The function you pass in will be called immediately after a hypothetical database write made where on_commit() is called would be successfully committed.

If you call on_commit() while there isn't an active transaction, the callback will be executed immediately.

If that hypothetical database write is instead rolled back (typically when an unhandled exception is raised in an atomic() block), your function will be discarded and never called.

Answer 4

I have come to complement the good response of @Vignesh

The example of the problem would be solved like this:

from django.db import transaction

def finalize_data(data):
    data = update_statistics(data)
    data.save()
    from apps.datas.tasks import optimize_data
    transaction.on_commit(lambda: optimize_data.delay(data.pk)) <---- Here

Does anymore else think that is the rigth answer? Or are there other clearer and optimized responses? Thanks