Django how to make this view faster?

Question

I have view. It takes about 1000 records from model and calculate for each two values. It works correct, but very slow about 1 minute.

My model. It contain readings for each day:

class Reading(models.Model):
    meter = models.ForeignKey(Meter, verbose_name=_('meter'))
    reading = models.FloatField(verbose_name=_('reading'))
    code = models.ForeignKey(ReadingCode, verbose_name=_('code'))
    date = models.DateTimeField(verbose_name=_('date'))

    class Meta:
        get_latest_by = 'date'
        ordering = ['-date', ]

    def __unicode__(self):
        return u'%s' % (self.date,)

    @property
    def consumption(self):
        try:
            end = self.get_next_by_date(code=self.code, meter=self.meter)
            return (end.reading - self.reading) / (end.date - self.date).days
        except:
            return 0.0

    @property
    def middle_consumption(self):
        data = []
        current_year = self.date.year
        for year in range(current_year - 3, current_year):
            date = datetime.date(year, self.date.month, self.date.day)
            try:
                data.append(Reading.objects.get(
                    date = date,
                    meter = self.meter,
                    code = self.code
                ).consumption)
            except:
                data.append(0.0)
            for i in data:
                if not i:
                    data.pop(0)
        return sum(data) / len(data)

My view. It returns json with all readings for requested meter and with calculated consumption and calculated middle consumption for last 3 years.

class DataForDayChart(TemplateView):
    def get(self, request, *args, **kwargs):
        output = []
        meter = Meter.objects.get(slug=kwargs['slug'])
        # TODO: Make it faster
        for reading in meter.readings_for_period().order_by('date'):
            output.append({
                "label": reading.date.strftime("%d.%m.%Y"),
                "reading": reading.reading,
                "value": reading.consumption / 1000,
                "middle": reading.middle_consumption / 1000
            })
        return HttpResponse(output, mimetype='application/json')

What should I change to make it faster?

Answer 1

The performance issue may caused by too many db operations, eg in method middle_consumption you query db at least twice,

 end = self.get_next_by_date(code=self.code, meter=self.meter)
 ...
 data.append(Reading.objects.get(
                date = date,
                meter = self.meter,
                code = self.code
            ).consumption)

You didn't show the all codes, so i suppose each step in following loop needs sql queries.

for reading in meter.readings_for_period().order_by('date'):

And as you said, there is only 1000 records, maybe you can load the data once and manipulate the relations and calculating in memory, which should improve the overall performance.

Answer 2

From the view name, I would imagine it's data does not change much during the day; in this case, I would recommend you to use caching.
Django has a good caching framework which is quite easy and straightforward to setup and use, and it will immediately make a huge difference, without much effort.

---

Of course, the first call will still be slow; and there, you may want to optimize it.
The usual approach in optimization is, first and foremost, measure, and you should profile the view, to see what are the slowest functions.
Alternatively, you could also insert some print statement to get a gist of where the slowdown occurs; the advantage here, is that you do not need to learn how to use a new tool.

That said, my best guess is that the slowdown happens in the call to meter.readings_for_period() (whose code you did not post), and is due to some inefficient database query - for instance, eventually instructing the ORM to retrieve the records one by one instead that with a single select statement.