Django: how do you deal with queries that can not scale?

Question

In my code, I have a few queries that are not scalable at all.

For example, take a look at this code:

class OrderQuerySet(query.QuerySet):
    def for_day(self, day: date):
        """ Return all orders that concerns the given service day """

        day_order_pks = [order.pk for order in self.all()
                         if localdate(order.service.start) == day]
        return self.filter(pk__in=day_order_pks)

At the beginning, it perfectly worked. The problem is, when order amount increase, the performance seems to decreases in a linear way, which makes senses because all orders need to be tested each time. Having 1000 new orders every day, it's obvious my system will not be usable anymore in a few time!

Usually, how do you deal with this kind of problem in Django?

I mean, sometimes I can find a trick to write a better query, using Django ORM only. But sometimes, to get what I want, I seems to be forced to create my queryset this way, using Python and a for loop.

Answer 1

Please don't enumerate over .all() unless you absolutely have to. It is more efficient to do filtering at the database side. Given localdate(..) does not do much, except extracting the date from a datetime , you can filter with:

class OrderQuerySet(query.QuerySet):

    def for_day(self, day: date):
        """ Return all orders that concerns the given service day """
        return self.

If localdate(..) is more advanced, you can still try to do most of the work at the database side. For example by filtering the queryset down to orders that are within for example 24 hours of the given date , and then do advanced filtering at the Python/Django side. But the idea is to do as much as possible at the database side (unless you make some exotic queries that do not scale well at the database, but that is quite rare).