Django development server reload takes too long

Question

This has been my problem since I've upgraded to OSX Lion: Whenever the runserver reloads when I change a file in my Django project, it takes quite a while before it starts serving again.

This happens even in a newly created Django 1.4 project. Didn't have this problem though on Snow Leopard.

I used cProfile and this is where it spent most of its time:

Ordered by: cumulative time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.001    0.001   48.068   48.068 manage.py:2(<module>)
    1    0.000    0.000   48.033   48.033 __init__.py:431(execute_manager)
    1    0.000    0.000   48.032   48.032 __init__.py:340(execute)
    1    0.000    0.000   47.908   47.908 base.py:182(run_from_argv)
    1    0.000    0.000   47.907   47.907 base.py:193(execute)
    1    0.000    0.000   47.814   47.814 runserver.py:39(handle)
    1    0.000    0.000   47.814   47.814 runserver.py:69(run)
    1    0.001    0.001   47.814   47.814 autoreload.py:129(main)
    1    0.000    0.000   47.813   47.813 autoreload.py:107(python_reloader)
    1    0.000    0.000   47.813   47.813 autoreload.py:96(restart_with_reloader)
    1    0.000    0.000   47.813   47.813 os.py:565(spawnve)
    1    0.000    0.000   47.813   47.813 os.py:529(_spawnvef)
    1   47.812   47.812   47.812   47.812 {posix.waitpid}
    ...

But I don't understand why?

Answer 1

(for guys still googling the answer)

I had similar problem using Vagrant (on Windows host machine). Solution for me was move virtualenv folder away from synced /vagrant . Default settings of synced folders uses VirtualBox provider and that's the problem. We can read about this in another sync methods from Vagrant official documentation :

In some cases the default shared folder implementations (such as VirtualBox shared folders) have high performance penalties. If you're seeing less than ideal performance with synced folders, NFS can offer a solution.

and

SMB is built-in to Windows machines and provides a higher performance alternative to some other mechanisms such as VirtualBox shared folders.

See Vagrant shared folders benchmark for extra info.

Answer 2

The manpage for waitpid says: The waitpid() system call suspends execution of the calling process until a child specified by pid argument has changed state. By default, waitpid() waits only for terminated children, but this behavior is modifiable via the options argument, as described below. http://linux.die.net/man/2/waitpid

Why is it taking so long for the child process to change state? The django manage.py runserver command is a very thin wrapper over ANOTHER runserver command:

 2533 pts/0    Ss     0:00  \_ bash
28374 pts/0    S+     0:00  |   \_ ../env/bin/python ./manage.py runserver
 7968 pts/0    Sl+   20:26  |       \_/home/sandford/workspace/usgm_apps/usgm_apps/../env/bin/python ./manage.py runserver

So the "boss" (28374) notices a change on a file and tells the "worker" (7968) to exit. Once the "worker" exits, it starts up a new worker with the new source files. The "worker" is taking a long time to exit.

Or OSX THINKS it's taking a long time to exit. If the OS gets behind on bookkeeping in the kernel for some reason and delays updating state you could end up with a delay like this.

Or perhaps there's something else entirely going on. It's perplexing.

Answer 3

In my case, it was caused by the DjangoWhiteNoise module being loaded in the wsgi.py file. After I added a condition that disabled the module in my development environment, the server reload time decreased substantially.