Running Thread even after the main thread terminates using Django

Question

I am working on a django project where the user hits a submits a bunch of images that needs to be processed in the backend.

The images are already uploaded on the server but once the user hits submit, it takes a lot of time to process the request and about 15 seconds until a 'thank you for using us' message is displayed.

What I wanted to do is to put the time consuming part of the process into a different thread and display the thank you message right away. My code looks like this:

def processJob(request):
    ...
    threading.Thread(target=processInBackground, args=(username, jobID)).start()
context = {}
context.update(csrf(request))
return render_to_response('checkout.html', context)

def processInBackground(username, jobID):
...
    (processing the rest of the job)

However, once I run it: It create a new thread but it terminates the seconds the main thread terminates. Is there any way how I can process the stuff in the backend, while the user gets the thank message right away?

Answer 1

PROCESSORS = [] # empty list of background processors

Inherit from Trheading.thread

class ImgProcessor(threading.Thread):
    def __init__(self, img, username, jobID):
        self.img = img
        self.username = username
        self.jobID = jobID
        threading.Thread.__init__(self)
        self.start()
        self.readyflag = False

    def run(self):
        ... process the image ...
        self.readyflag = True

Then, when receiving the request:

def processJob(request):
    PROCESSORS.append(ImgProcessor(img, username, jobID))
    .... remove all objects in PROCESSORS that have readyflag == True

Answer 2

This isn't necessarily what you're looking for, but with my photographer website platform we upload the photo asynchronously using Javascript (with PLUpload), which allows us to have callbacks for multiple photo upload statuses at a time. Once uploaded, the file is saved out to a folder with a unique name, and also into a database queue, where it is then picked up by a cron job that runs through the queue for anything that's not yet completed, and processes it. I am using Django's custom management commands so I get all the benefit of the Django framework, minus the web part.

Benefits of this are that you get a record of every upload, you can display status with a polling request, and you can perform the processing on any server that you wish if you'd like.