GAE background Threads not running in parallel

Question

I need some help figuring out a problem with GAE background threads. What I'm trying to do is run a job on several threads using GAE background threads and Queue. The code runs on a Backend instance, and is kicked of by a TaskQueue. What I'm getting is that the jobs in the threads are running serially instead of in parallel, which sort of defeats the purpose.

from Queue import Queue
from google.appengine.api import background_thread
from google.appengine.api import taskqueue

q = Queue()

class Util():
   def work_in_background_thread(self):

       for p in portions:
            q.put(p)

        def _worker(index):
            portion = q.get()                
            do_work(portion)
            q.task_done()


        def do_work(snp_list):
            for snp in snp_list:
                self.find_snp_data(snp)


        for i in range(len(portions)):
            try:
                t = background_thread.BackgroundThread(target=_worker, args=[i])
                t.setDaemon(True)
                t.start()
            except:
                continue

        q.join()  

The work is getting done, but threads run one after the other, so this job is taking hours to run!

Answer 1

If you are using a backend then it's being run on a single instance and on the dev server it is impossible to get concurrency due to python Global Interpreter Lock (GIL) . I suspect that might be you issue. Also take a look here to understand that python cannot handle concurrency, but can handle threads to deliver better I/O

EDIT

Today there has been a new release of dev_server: Quoting from the blog :

A major overhaul to the Python dev_appserver, the software used to simulate App Engine while in development. The new dev_appserver is multi-threaded, meaning development is faster, and provides a more accurate simulation of the production environment.