使用 INNER JOIN 和 SELECT DISTINCT ON 優化查詢的主鍵和索引

Question

我有一個dbfiddle 演示，其中定義了下表：

CREATE TABLE status_table (
  base_name  text      NOT NULL
, version    smallint  NOT NULL
, ref_time   int       NOT NULL
, processed  bool      NOT NULL
, processing bool      NOT NULL
, updated    int       NOT NULL
, PRIMARY KEY (base_name, version)
);

CREATE TABLE data_table (
  location  text      NOT NULL
, param_id  text      NOT NULL
, ref_time  int       NOT NULL
, fcst_time smallint  NOT NULL
, timestamp int       NOT NULL
, value     text      NOT NULL
, PRIMARY KEY (location, param_id, ref_time, fcst_time)
);

沒有定義其他索引。

請注意，對於data.table中的每一行，我知道ref_time + fcst_time = timestamp的情況並不理想，但這是它演變的方式。 所以ref_time就像一個基准時間（對於一批數據）而fcst_time是一個偏移時間，給出數據記錄的實際timestamp （每個批次都有一個數據記錄的時間序列，從ref_time開始並且具有逐漸增加的timestamp或fcst_time和單個數據value ）。

然后我有以下復雜的查詢，用於從data.table中刪除選定的行。 它從status_table中提取一些信息作為stats並將其加入data.table ，然后選擇不應刪除的行（ sel1和sel2 ），然后刪除data.table中不在sel1中也不在sel2中的所有行。

順便說一句， sel1基本上對應於我從data.table讀取數據的查詢（雖然我在這樣做時限制在特定location ，因此它非常快）......因此sel1只是可能在查詢...我想保留這些而不是刪除它們。

然后sel2是那些與仍在處理的數據相關的行，所以我也需要保留這些行。

因此，考慮到這一點，這里是查詢：

WITH
  stats AS (
    SELECT ref_time
      , max(updated) < (round(extract(epoch from now()) / 60) - 200) AS settled
      , (count(*) FILTER (WHERE processed) = count(*)) AND (max(updated) < (round(extract(epoch from now()) / 60) - 200)) AS ready
    FROM status_table
    GROUP BY ref_time
  ),
  min_ts AS (
    SELECT ref_time FROM stats WHERE ready ORDER BY ref_time DESC LIMIT 1
  ),
  sel1 AS (
    -- we need to keep all of these rows (don't delete)
    SELECT DISTINCT ON (d.location, d.timestamp, d.param_id)
      d.location, d.param_id, d.ref_time, d.fcst_time
    FROM data_table AS d
    INNER JOIN stats s USING (ref_time)
    WHERE s.ready AND d.timestamp >= (SELECT ref_time FROM min_ts)
    ORDER BY d.location, d.timestamp, d.param_id, d.ref_time DESC
  ),
  sel2 AS (
    -- we also need to keep all of these rows (don't delete)
    SELECT
      d.location, d.param_id, d.ref_time, d.fcst_time
    FROM data_table AS d
    INNER JOIN stats AS s USING (ref_time)
    WHERE NOT s.settled
  )
DELETE FROM data_table 
  WHERE
    (location, param_id, ref_time, fcst_time) NOT IN (SELECT location, param_id, ref_time, fcst_time FROM sel1)
  AND
    (location, param_id, ref_time, fcst_time) NOT IN (SELECT location, param_id, ref_time, fcst_time FROM sel2);

但我發現這在我的實際數據庫中非常慢。 我知道我需要優化我的索引和可能的主鍵，並且嘗試了各種方法但沒有任何真正的成功，所以我有點迷茫。

這是我實際數據庫中上述查詢的EXPLAIN的 output：

                                                       QUERY PLAN                                                        
-------------------------------------------------------------------------------------------------------------------------
 Delete on data_table  (cost=4002975.62..118180240066541.86 rows=0 width=0)
   CTE stats
     ->  HashAggregate  (cost=234.02..234.21 rows=4 width=6)
           Group Key: status_table.ref_time
           ->  Seq Scan on status_table  (cost=0.00..164.01 rows=7001 width=9)
   ->  Seq Scan on data_table  (cost=4002741.41..118180240066307.66 rows=19567628 width=6)
         Filter: ((NOT (SubPlan 3)) AND (NOT (SubPlan 4)))
         SubPlan 3
           ->  Materialize  (cost=4002741.30..4293628.93 rows=7691318 width=18)
                 ->  Subquery Scan on sel1  (cost=4002741.30..4210105.34 rows=7691318 width=18)
                       ->  Unique  (cost=4002741.30..4133192.16 rows=7691318 width=22)
                             InitPlan 2 (returns $1)
                               ->  Limit  (cost=0.09..0.09 rows=1 width=4)
                                     ->  Sort  (cost=0.09..0.10 rows=2 width=4)
                                           Sort Key: stats.ref_time DESC
                                           ->  CTE Scan on stats  (cost=0.00..0.08 rows=2 width=4)
                                                 Filter: ready
                             ->  Sort  (cost=4002741.20..4035353.91 rows=13045086 width=22)
                                   Sort Key: d.location, d."timestamp", d.param_id, d.ref_time DESC
                                   ->  Hash Join  (cost=0.11..1925948.51 rows=13045086 width=22)
                                         Hash Cond: (d.ref_time = s.ref_time)
                                         ->  Seq Scan on data_table d  (cost=0.00..1697659.40 rows=26090171 width=22)
                                               Filter: ("timestamp" >= $1)
                                         ->  Hash  (cost=0.08..0.08 rows=2 width=4)
                                               ->  CTE Scan on stats s  (cost=0.00..0.08 rows=2 width=4)
                                                     Filter: ready
         SubPlan 4
           ->  Materialize  (cost=0.11..2611835.48 rows=39135256 width=18)
                 ->  Hash Join  (cost=0.11..2186850.21 rows=39135256 width=18)
                       Hash Cond: (d_1.ref_time = s_1.ref_time)
                       ->  Seq Scan on data_table d_1  (cost=0.00..1501983.12 rows=78270512 width=18)
                       ->  Hash  (cost=0.08..0.08 rows=2 width=4)
                             ->  CTE Scan on stats s_1  (cost=0.00..0.08 rows=2 width=4)
                                   Filter: (NOT settled)
 JIT:
   Functions: 45
   Options: Inlining true, Optimization true, Expressions true, Deforming true
(37 rows)

Answer 1

這會改善你的解釋計划嗎？

工會刪除並檢查您的刪除

WITH
  stats AS (
    SELECT ref_time
      , max(updated) < (round(extract(epoch from now()) / 60) - 200) AS settled
      , (count(*) FILTER (WHERE processed) = count(*)) AND (max(updated) < (round(extract(epoch from now()) / 60) - 200)) AS ready
    FROM status_table
    GROUP BY ref_time
  ),
  min_ts AS (
    SELECT ref_time FROM stats WHERE ready ORDER BY ref_time DESC LIMIT 1
  ),
  sel1 AS (
    -- records that would be selected by an actual data lookup (use same logic)... we need to keep these (don't delete)
    SELECT DISTINCT ON (d.location, d.timestamp, d.param_id)
      d.location, d.param_id, d.ref_time, d.fcst_time
    FROM data_table AS d
    INNER JOIN stats s USING (ref_time)
    WHERE s.ready AND d.timestamp >= (SELECT ref_time FROM min_ts)
    ORDER BY d.location, d.timestamp, d.param_id, d.ref_time DESC
  ),
  sel2 AS (
    -- also keep all records that are in-progress (not 'settled')
    SELECT
      d.location, d.param_id, d.ref_time, d.fcst_time
    FROM data_table AS d
    INNER JOIN stats AS s USING (ref_time)
    WHERE NOT s.settled
  ),
  sel AS (
    SELECT * FROM sel1
    UNION SELECT * FROM sel2
  )
DELETE FROM data_table 
  WHERE
    (location, param_id, ref_time, fcst_time) NOT IN (SELECT location, param_id, ref_time, fcst_time FROM sel);