Django 过滤大型数据集耗时太长

Question

I know there have been a lot of questions regarding this subject, but I couldn't find any of my case. My problem is rather simple.

In my influencer app, I have Note model which contains about 30 fields:

class Note(models.Model):
    desc = models.TextField()
    likeCount = models.PositiveIntegerField()
    commentCount = models.PositiveIntegerField()
    ...
    tags = models.ManyToManyField(Tag)
    postTs = models.DateTimeField(null=True)

And there is more than 1 million Note s in my PostgreSQL database hosted by AWS RDS.

Now when I execute the following code:

notes = (
    Note.objects.filter(desc__icontains='some word')
                .values("likeCount", "collectCount", "shareCount", "commentCount", "postTs")[:10]
)
print(len(notes))  # Output: 10

it takes around 7 seconds .

The resulting SQL query is:

SELECT "influencer_note"."likeCount",
       "influencer_note"."collectCount",
       "influencer_note"."shareCount",
       "influencer_note"."commentCount",
       "influencer_note"."postTs"
  FROM "influencer_note"
 WHERE UPPER("influencer_note"."desc"::text) LIKE UPPER('%some word%')
 LIMIT 10

I think I have done pretty much everything to optimize the query (such as selecting the only necessary fields and limiting the number of data -- 10 is obviously a small number ), but it's still taking abnormal amount of time.

What are the possible causes for this problem and how can I further optimize this?

Ultimately, I need to make a chart with the filtered queryset, which is why I need a solution other than pagination or LIMIT .

Thank you in advance.

Answer 1

It's very likely that icontains (LIKE UPPER) here is an expensive operation that is taking most of query evaluation time. Not sure if you can do much more optimizations with Django ORM, but probably you can try some of the approaches of full text search with using PostgreSQL vector search.

Another option is to use more suitable tool like ElasticSearch. You can read some entrance guide here .

Answer 2

The query you show could be accelerated by the index:

create extension pg_trgm;
create index on influencer_note using gin (UPPER("desc"::text) gin_trgm_ops)

Although why Django injects the UPPER calls in the query here, rather than just doing the sensible thing of using ILIKE, is a mystery to me.