Django model doesn't get saved to database inside Celery Task

Question

I've hit a really nasty situation. I have the following setup. I have a django model representing an FSM with a django FSM field

I have a celery task that sends out an email and then advances the state of the main objects FSM. From the celery task's perspective, the object "seems" to be saved. But from the main django process' perspective, the object isn't being updated. The strange thing is that ancillary objects are being saved properly to the DB, and later accessible from the main django process.

I explicitly call .save() on the object from the Celery task, and the date_last_modified = models.DateTimeField(auto_now=True, null=True) field has a later timestamp in the Celery task than the main thread, although I'm not sure if that's an indication of anything, ie it may have been updated but the update has not been flushed out to the DB.

I'm using django 1.5.1, postgresql 9.3.0, celery v3.1.0, Redis 2.6.10

Running Celery like so $ celery -A tracking worker -E -B -l info

Any ideas of why this may be happening would be greatly appreciated

Answer 1

Are you re-getting the object after the save? Ie not just looking at the instance you got before the save?

Answer 2

I had similar problem with Django 1.5

I guess it's because of that Django does not commit changes to database immediately.

Adding

    'OPTIONS': {
        'autocommit': True
    }

to DATABASES setting fixed the problem for me.

Problem will not exist in Django 1.6+ beacuse autocommit is the default there.

Answer 3

What about transactions? You can try to set CELERY_EAGER_PROPAGATES_EXCEPTIONS=True and run celery with -l DEBUG to see, is any error happens after model .save() call.

Also take care of concurrent updates. When one process reads model, then celery reads and saves same model, if initial process calls models.save() later it would override all fields in it.

Answer 4

Had an issue looking like yours on Django 3.2.7. using get_nb_lines.delay(flow.pk) within an class based updateview.

After fix, I suppose it was a kind of (maybe) concurrent updates or crossing updates (dunno how to call that).

I understood that after I noticed that get_nb_lines.apply_async((flow.pk,), countdown=5) had fixed my problem. I anybody explains this another way, I'll take it.

Take care because the parameter sent into the function must be iterable as said an error alert. So in my case, I had to treat flow.pk as a list (add a comma after flow.pk)