Django: SQL 由於復雜的子查詢，查詢速度很慢，如何拆分或加快它們的速度？

Question

我需要執行一個帶有復雜子查詢的查詢集，如下所示。 執行查詢需要相當長的時間。 (8000 毫秒)

我認為減速是由復雜的子查詢引起的，但是是否可以在不生成 N+1 的情況下拆分或加速單個 SQL 查詢？

我們使用的db查找和這次設置的慢查詢

# lookups.py
class Groonga(Lookup):
    lookup_name = "groonga"

    def as_sql(self, compiler, connection):
        lhs, lhs_params = self.process_lhs(compiler, connection)
        rhs, rhs_params = self.process_rhs(compiler, connection)
        params = lhs_params + rhs_params
        return "%s &@~ %s" % (lhs, rhs), params

# queryset
Video.objects.annotate(
    is_viewed=Exists(History.objects.filter(user=user, video=OuterRef("pk"))),
    is_favorited=Exists(
        Favorite.objects.filter(user=user, video=OuterRef("pk"))
    ),
    is_wl=Exists(
        Track.objects.filter(
            playlist__user=user, playlist__is_wl=True, video=OuterRef("pk")
        )
    ),
).filter(
    Q(title__groonga=value)
    | Q(tags__pk__in=Tag.objects.filter(name__groonga=value).values_list("pk")),
    is_public=True,
    published_at__lte=timezone.now(),
).order_by("-published_at").distinct()[:20]

SQL 查詢和解釋分析結果

SELECT DISTINCT "videos_video"."id",
                "videos_video"."published_at",
                EXISTS
  (SELECT (1) AS "a"
   FROM "videos_history" U0
   WHERE (U0."user_id" IS NULL
          AND U0."video_id" = "videos_video"."id")
   LIMIT 1) AS "is_viewed",
                EXISTS
  (SELECT (1) AS "a"
   FROM "videos_favorite" U0
   WHERE (U0."user_id" IS NULL
          AND U0."video_id" = "videos_video"."id")
   LIMIT 1) AS "is_favorited",
                EXISTS
  (SELECT (1) AS "a"
   FROM "videos_track" U0
   INNER JOIN "videos_playlist" U1 ON (U0."playlist_id" = U1."id")
   WHERE (U1."is_wl"
          AND U1."user_id" IS NULL
          AND U0."video_id" = "videos_video"."id")
   LIMIT 1) AS "is_wl"
FROM "videos_video"
LEFT OUTER JOIN "videos_video_tags" ON ("videos_video"."id" = "videos_video_tags"."video_id")
WHERE ("videos_video"."is_public"
       AND "videos_video"."published_at" <= '2021-12-27 13:34:29.103369+00:00'
       AND ("videos_video"."title" &@~ 'word'
            OR "videos_video_tags"."tag_id" IN
              (SELECT U0."id"
               FROM "videos_tag" U0
               WHERE U0."name" &@~ 'word')))
ORDER BY "videos_video"."published_at" DESC
LIMIT 20;

--                                                                                QUERY PLAN                               
-- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--  Limit  (cost=25920044.10..25920044.40 rows=20 width=27) (actual time=8031.477..8083.885 rows=20 loops=1)
--    ->  Unique  (cost=25920044.10..25930094.65 rows=670037 width=27) (actual time=7498.770..7551.175 rows=20 loops=1)
--          ->  Sort  (cost=25920044.10..25921719.19 rows=670037 width=27) (actual time=7498.768..7498.796 rows=30 loops=1)
--                Sort Key: videos_video.published_at DESC, videos_video.id, ((hashed SubPlan 2)), ((hashed SubPlan 4)), ((hashed SubPlan 6))
--                Sort Method: external merge  Disk: 12232kB
--                ->  Hash Right Join  (cost=22049.49..25839171.52 rows=670037 width=27) (actual time=258.201..7221.055 rows=336302 loops=1)
--                      Hash Cond: (videos_video_tags.video_id = videos_video.id)
--                      Filter: ((videos_video.title &@~ 'word'::character varying) OR (hashed SubPlan 7))
--                      Rows Removed by Filter: 1002707
--                      ->  Seq Scan on videos_video_tags  (cost=0.00..24544.40 rows=1338740 width=32) (actual time=0.071..321.529 rows=1338740 loops=1)
--                      ->  Hash  (cost=13230.75..13230.75 rows=290059 width=117) (actual time=229.084..229.085 rows=290059 loops=1)
--                            Buckets: 32768  Batches: 16  Memory Usage: 2899kB
--                            ->  Seq Scan on videos_video  (cost=0.00..13230.75 rows=290059 width=117) (actual time=0.049..80.893 rows=290059 loops=1)
--                                  Filter: (is_public AND (published_at <= '2021-12-27 13:34:29.103369+00'::timestamp with time zone))
--                                  Rows Removed by Filter: 1
--                      SubPlan 2
--                        ->  Bitmap Heap Scan on videos_history u0  (cost=4.18..12.63 rows=4 width=16) (actual time=0.005..0.007 rows=0 loops=1)
--                              Recheck Cond: (user_id IS NULL)
--                              ->  Bitmap Index Scan on videos_history_user_id_9a1343c1  (cost=0.00..4.18 rows=4 width=0) (actual time=0.005..0.005 rows=0 loops=1)
--                                    Index Cond: (user_id IS NULL)
--                      SubPlan 4
--                        ->  Bitmap Heap Scan on videos_favorite u0_1  (cost=4.19..12.65 rows=5 width=16) (actual time=0.003..0.004 rows=0 loops=1)
--                              Recheck Cond: (user_id IS NULL)
--                              ->  Bitmap Index Scan on videos_favorite_user_id_c4289dec  (cost=0.00..4.19 rows=5 width=0) (actual time=0.002..0.003 rows=0 loops=1)
--                                    Index Cond: (user_id IS NULL)
--                      SubPlan 6
--                        ->  Nested Loop  (cost=8.36..23.98 rows=2 width=16) (actual time=0.059..0.061 rows=0 loops=1)
--                              ->  Bitmap Heap Scan on videos_playlist u1  (cost=4.17..11.27 rows=1 width=16) (actual time=0.058..0.059 rows=0 loops=1)
--                                    Recheck Cond: (user_id IS NULL)
--                                    Filter: is_wl
--                                    ->  Bitmap Index Scan on videos_playlist_user_id_e71a2f32  (cost=0.00..4.17 rows=3 width=0) (actual time=0.054..0.054 rows=0 loops=1)
--                                          Index Cond: (user_id IS NULL)
--                              ->  Bitmap Heap Scan on videos_track u0_2  (cost=4.19..12.66 rows=5 width=32) (never executed)
--                                    Recheck Cond: (playlist_id = u1.id)
--                                    ->  Bitmap Index Scan on videos_track_playlist_id_bfcae4d7  (cost=0.00..4.19 rows=5 width=0) (never executed)
--                                          Index Cond: (playlist_id = u1.id)
--                      SubPlan 7
--                        ->  Bitmap Heap Scan on videos_tag u0_3  (cost=0.00..93.99 rows=6 width=16) (actual time=26.298..26.322 rows=18 loops=1)
--                              Recheck Cond: (name &@~ 'word'::character varying)
--                              Heap Blocks: exact=11
--                              ->  Bitmap Index Scan on pgroonga_tag_index  (cost=0.00..0.00 rows=56 width=0) (actual time=0.611..0.612 rows=18 loops=1)
--                                    Index Cond: (name &@~ 'word'::character varying)
--  Planning Time: 3.298 ms
--  JIT:
--    Functions: 75
--    Options: Inlining true, Optimization true, Expressions true, Deforming true
--    Timing: Generation 14.381 ms, Inlining 20.362 ms, Optimization 374.869 ms, Emission 215.556 ms, Total 625.169 ms
--  Execution Time: 8103.397 ms
-- (48 rows)

-- Time: 8108.641 ms (00:08.109)

Answer 1

您可能有復雜的子查詢，但這不是您需要更好執行的第一次攻擊。 這是罪魁禍首： Sort Method: external merge Disk: 12232kB 。 您正在執行磁盤排序而不是 memory。 你需要增加work_mem。

work_mem（整數）

在寫入臨時磁盤文件之前，設置查詢操作 >（例如排序或 hash 表）使用的 memory 的基本最大數量。