在Django视图中保持状态以提高分页性能

Question

I'm designing a data-tables -driven Django app and have an API view that data-tables calls with AJAX (I'm using data-tables in its server-side processing mode). It implements searching, pagination, and ordering.

My database recently got large (about 500,000 entries) and performance has greatly suffered, both for searches and for simply moving to the next page. I suspect that the way I wrote the view is grossly inefficient. Here's what I do in the view (suppose the objects in my database are pizzas):

• filtered = Pizza.objects.filter(...) to get the set of pizzas that match the search criteria. (Or Pizza.objects.all() if there is no search criteria).

• paginated = filtered[start: start + length] to get only the current page of pizzas. (At max, only 100 of them). Start and length are passed in from the data-tables client-side code, according to what page the user is on.

• pizzas = paginated.order_by(...) to apply the ordering to the current page.

Then I convert pizzas into JSON and return them from the view.

It seems that, while search might justifiably be a slow operation on 500,000 entries, simply moving to the next page shouldn't require us to redo the whole search. So what I was thinking of doing was caching some stuff in the view (it's a class-based view). I would keep track of what the last search string was, along with the set of results it produced.

Then, if a request comes through and the search string isn't different (which is what happens if the user is clicking through a few pages of results) I don't have to hit the database again to get the filtered results -- I can just use the cached version.

It's a read-only application, so getting out of sync would not be an issue.

I could even keep a dictionary of a whole bunch of search strings and the pizzas they should produce.

What I'd like to know is : is this a reasonable solution to the problem? Or is there something I'm overlooking? Also, am I re-inventing the wheel here? Not that this wouldn't be easy to implement, but is there a built-in option on QuerySet or something to do this?

Answer 1

pizzas = paginated.order_by(...) is slow, it sorts all Pizzas NOT the current page. Indexes help: https://docs.djangoproject.com/en/1.8/topics/db/optimization/#use-standard-db-optimization-techniques

If you really want cache, checkout https://github.com/Suor/django-cacheops , "A slick app that supports automatic or manual queryset caching and automatic granular event-driven invalidation."

Answer 2

There are multiple way of improving your code structure,

First is you fetch only that data which is required according to your page number using Django ORM hit, second is you cache your ORM output and reuse that result if same query is passed again.

First goes like this.

In your code

Pizza.objects.all() paginated = filtered[start: start + length] You are first fetching all data then, you are slicing that, which is very expensive SQL query, convert that to

filtered = Pizza.objects.all()[(page_number-1) * 30, (page_number-1) * 30 + 30]

above given ORM will only fetch those rows which are according to supplied page number and is very fast compare to fetching all and then slicing it.

The second way , is you first fetch data according to query put that on, caching solution like memcache or redis, next time when you are required to fetch the data from the database, then first check if data is present in cache for that query, if present, then simply use that data, because in-memory caching solution are way faster than fetching the data from the database because of very large input output transfer between memory and hard drive and we know hard drives are traditionally slow.