django/python : error when get value from dictionary

Question

I have python/django code hosted at dotcloud and redhat openshift. For handling different user, I use token and save it in dictionary. But when I get the value from dict, it sometimes throws an error(key value error).

import threading

thread_queue = {}

def download(request):
    dl_val = request.POST["input1"]
    client_token = str(request.POST["pagecookie"])
        # save client token as keys and thread object as value in dictionary
    thread_queue[client_token] = DownloadThread(dl_val,client_token)
    thread_queue[client_token].start()
    return render_to_response("progress.html",
              { "dl_val" : dl_val, "token" :      client_token })

The code below is executed in 1 second intervals via javascript xmlhttprequest to server. It will check variable inside another thread and return the value to user page.

def downloadProgress(request, token):
        # sometimes i use this for check the content of dict
    #resp = HttpResponse("thread_queue = "+str(thread_queue))
    #return resp
    prog, total = thread_queue[str(token)].getValue() # problematic line !
    if prog == 0:
                # prevent division by zero
        return HttpResponse("0")
    percent = float(prog) / float(total)
    percent = round(percent*100, 2)
    if percent >= 100:
        try:
            f_name = thread_queue[token].getFileName()[1]
        except:
            downloadProgress(request,token)
        resp = HttpResponse('<a href="http://'+request.META['HTTP_HOST']+
                            '/dl/'+token+'/">'+f_name+'</a><br />')
        return resp
    else:
        return HttpResponse(str(percent))

After testing for several days, it sometimes return :

thread_queue = {}

It sometimes succeeds :

thread_queue = {'wFVdMDF9a2qSQCAXi7za': , 'EVukb7QdNdDgCf2ZtVSw': , 'C7pkqYRvRadTfEce5j2b': , '2xPFhR6wm9bs9BEQNfdd': } 

I never get this result when I'm running django locally via manage.py runserver, and accessing it with google chrome, but when I upload it to dotcloud or openshift, it always gives the above problem. My question :

• How can I solve this problem ?
• Does dotcloud and openshift limit their python cpu usage ?
• Or is the problem inside the python dictionary ?

Thank You.

Answer 1

dotCloud has 4 worker processes by default for the python service. When you run the dev server locally, you are only running one process. Like @martijn said, your issue is related to the fact that your dict isn't going to be shared between these processes.

To fix this issue, you could use something like redis or memcached to store this information instead. If you need a more long term storage solution, then using a database is probably better suited.

dotCloud does not limit the CPU usage, The CPU is shared amongst others on the same host, and allows bursting, but in the end everyone has the same amount of CPU.

Looking at your code, you should check to make sure there is a value in the dict before you access it, or at a minimum surround the code with a try except block, to handle the case when the data isn't there.

str_token = str(token)
if str_token in thread_queue:
   prog, total = thread_queue[str_token].getValue() # problematic line !
else:
   # value isn't there, do something else 

Answer 2

Presumably dotcloud and openshift run multiple processes of your code; the dict is not going to be shared between these processes.

Note that that also means the extra processes will not have access to your extra tread either.

Use an external database for this kind of information instead. For long-running asynchronous jobs like these you also need to run them in a separate worker process. Look at Celery for an all-in-one solution for asynchronous job handling, for example.