带有线程的随机python模块加载失败

Question

I'm trying to debug an error where python import statements randomly fail, at other times they run cleanly.

This is an example of the exceptions I see. Sometimes I'll see this one, sometimes I'll see another one in a different module, though it seems to always hit in one of 4 modules.

ERROR:root:/home/user/projecteat/django/contrib/auth/management/__init__.py:25:     RuntimeWarning: Parent module 'django.contrib.auth.management' not found while handling absolute import
from django.contrib.contenttypes.models import ContentType

Because of the random nature, I'm almost certain it's a threading issue, but I don't understand why I would get import errors, so I'm not sure what to look for in debugging. Can this be caused by filesystem contention if different threads are trying to load the same modules?

I'm trying to get Django 1.4's LiveServerTestCase working on Google App Engine's development server. The main thread runs django's test framework. When it loads up a LiveServerTestCase based test class, it spawns a child thread which launches the App Engine dev_appserver, which is a local webserver. The main thread continues to run the test, using the Selenium driver to make HTTP requests, which are handled by dev_appserver on the child thread.

The test framework may run a few tests in the LiveServerTestCase based class before tearing down the testcase class. At teardown, the child thread is ended.

It looks like the exceptions are happening in the child (HTTP server) thread, mostly between tests within a single testcase class.

The code for the App Engine LiveServerTestCase class is here: https://github.com/dragonx/djangoappengine/blob/django-1.4/test.py

It's pretty hard to provide all the debugging info required for this question. I'm mostly looking for suggestions as to why python import statements would give RuntimeWarning errors.

Answer 1

I have a partial answer to my own question. What's going on is that I have two threads running.

Thread 1 is running the main internal function inside dev_appserver (dev_appserver_main) which is handling HTTP requests.

Thread 2 is running the Selenium based testcases. This thread will send commands to the browser to do something (which then indirectly generates an HTTP request and re-enters in thread 1). It then either issues more requests to Selenium to check status, or makes a datastore query to check for a result.

I think the problem is that upon handling every HTTP request, Thread 1 (dev_appserver) changes the environment so that certain folders are not accessible (folder excluded in app.yaml, as well as the environment that is not part of appengine). If Thread 2 happens to run some code in this time, certain imports may fail to load if they are located in these folders.