How do I keep a shared data structure in-memory between Django Requests

Question

My Django application is extremely performance sensitive and all requests require access to the same data structure. How do I store the data structure in such a way that it is accessible to all the requests?

Background:

I'm currently using the cache backend. This is a bit slow because the DS is large and it has to be retrieved and unpickled each time.

I understand that HTTP interactions should be stateless and knowingly need to break this constraint. Nothing bad should happen because it's read-only right?

Answer 1

There are a several ways to deal with this issue:

1. Move the data structure out of Python completely (rather than loading it from a storage medium every time). For example, if your structure is conducive to it you could store it in Redis , MongoDB , Riak , or Neo4j . (As a bonus, you get the ability to query the data, if you need that ability).
2. Move the structure to a separate process and communicate with it using a pipe or queue .
3. Use a memory mapped file to share the data.

HTTP is stateless, but that doesn't mean that you can't preserve state between requests. You just have to do the work yourself (at the application level). The protocol doesn't do it for you. Ideally you avoid the state, as it makes it easier to scale horizontally, but not every application is easy to scale

Answer 2

Django, and probably the majority of web applications, use caching . Of course the efficacy of caching depends on how you use it, ie by storing the data retrieved most frequently.

Answer 3

Really useful and informative article on caching in Django here . Gives quantified speed improvements. Amazing how much faster you can get with a little tweaking.