How to design cleanup tasks for sorl-thumbnail in django?

Question

I'm using sorl-thumbnail which provides cached thumbnails of a given image. I want to delete generated thumbs when the origin image changes or is deleted. I know that sorl-thumbnail provides a delete method for that case, but..

• it won't be triggered through django-admin
• might be bad cause of race-conditions
• when using a kv store like redis - all data (file connections) might be gone (eg reboot)

What fits better is the sorl cleanup management command. This command will delete kv entries with broken origin link and the associated physical thumbs. But what if my kv storage is not complete? I don't want to have dead files lying around.

I thought about a celery task which will trigger the cleanup command and additionally checks all cached files, but don't know how to design that.

What's the best way to detect and cleanup that mess?

Answer 1

You can add a method clear_thumbnails() to your model. In the clear_thumbnails() you use the delete function from sorl-thumbnail to delete the existing thumbnails. If the next time a view is rendered with the {% thumbnail %} template tag is rendered, the new thumbnails are created.

Now you add a function that is run when the pre_save signal of the model that is related to the thumbnail is called. In this function you simple call your clear_thumbnails()

With this approach the thumbnails will be also deleted when you change the model in Django admin. (You could also catch the pre_delete signal)

Something like this:

## models.py ##
from django.db import models
from sorl.thumbnail import delete as delete_thumbnails

class Route(models.Model):
    ...
    def clear_thumbnails(self):
        delete_thumbnails(self.image)

## receivers.py ##
from django.db.models.signals import pre_save

@receiver(pre_save, sender=YourModel)
def receive_yourmodel_pre_save(sender, **kwargs):
    sender.clear_thumbnails()

Now you have a model that deletes the thumbnails whenever the model is changed. (Maybe you should add a check into the receiver if the of the pre_save or post_save signal if the image field was changed and only delete the thumbnails if this was happening)

If you have a lot of thumbnails then the management commands of sorl-thumbnails are useless. I have a few million entries in sorl-thumbnails key value store and the management commands never finish.

I ended up writing my own little cleanup script that is better then the one provided by sorl-thumbnails because it does not use so much memory and also shows a progress. Something like this:

function clean_thumbnails(self):
    max_id = MyModel.objects.all().order_by('-pk')[0].pk

    for id in xrange(max_id, 0, -1):
        try:
            my_obj = MyModel.objects.get(pk=id)
            my_obj.clear_thumbnails()
            print "processed object with id %s" % id
        except MyModel.DoesNotExist:
            pass

I have this cleanup script setup as a RunScript as defined the Django extensions . Check out the Django extensions and the RunScript part. This is quite handy.

Hope this helps.