在Django中加快大型QuerySet的迭代速度

Question

I'm trying my hand at the Riot API challenge , and I'm trying to use Django as a backend hosted on PythonAnywhere.com .

I have setup a database which uses a structure similar to the one below

class MatchDetails(models.Model):
    # Data fields

class Participant(models.Model):
    match = models.ForeignKey(MatchDetails)
    # Data fields

class Timeline(models.Model):
    participant = models.ForeignKey(Participant)
    # Data fields

# More fields, most with MatchDetails as foreign key.

I have written a function which retrieves and stores data, and I now have almost 40 000 games stored, with 10 participants in each game. My goal is to extract some statistics from this data, and I basically do something like:

allMatches = MatchDetails.objects.all()
for m in allMatches:
    participants = m.participant_set.all()
    for p in participants:
        # Increment some values
# save the result to the database

Currently it takes a little more than 2 hours.

2015-04-11 03:47:35 -- Completed task, took 7942.00 seconds, return code was 0.

This is a ridiculous amount of time, isn't it? Is there some way for me to speed it up?

I've tried to use iterator, and I also tried using .value_list and .all.values() to iterate over, but I am unable to get objects related through foreign key this way.

How do I speed up iteration of a large dataset in Django

Is there any way for me to access my foreign key objects when using value_list? Or is there anything else I can do to speed it up? Any pointers would be appreciated.

Thanks for reading!

Answer 1

knbk's answer is great. You could also do your counting in the database. For instance, if you had a field on the participant model for time spent playing and you wanted the average time that participants spent playing, you could use something like

Participant.objects.all().aggregate(Avg('time_spent_playing'))

Have a look at the Django aggregation docs for more info.

Answer 2

The best optimization at this point is to use prefetch_related() :

allMatches = MatchDetails.objects.prefetch_related('participant_set')
for m in allMatches:
    for p in m.participant_set.all():
        # Increment some values
# save the result to the database

This reduces your number of queries from about 40 000 to 2.

Answer 3

You can try to speed it up using prefetch related .

Also, in your values_list, you can get just the properties you need of your related objects traversing like "foreign_key_relation_name__attribute".

Using "iterator()" is also a good way of improving speed if you only iterate once through the queryset.

How is your "save the result to the database" code? If you save all items in batch with update() instead of saving each one by one, you will improve your speed as well.

Answer 4

Depending on two factors - what it is you're updating, and if you're running Django 1.8 - you don't have to iterate through everything.

from django.db.models import F
m.participant_set.update(some_field=F('some_field')*10)

This would update all some_field in the participants model to be its current value times ten which would be orders of magnitude faster than iterating over all the rows and doing an update per row.

Worth having in mind is that if you have overridden the Participant.save()-method, it won't be called, and save signals wont be sent either.