堆栈内存溢出
  繁体   English   中英

分区扫描很慢

[英]Partition scanning very slow

 原文  2022-08-03 02:27:52  6   1    postgresql/ indexing/ partitioning

问题描述

我有一个大表，分为两个级别，首先是时间：

CREATE TABLE IF NOT EXISTS research.ratios
(
    security_id integer NOT NULL,
    period_id smallint NOT NULL,
    classification_id smallint NOT NULL,
    dtz timestamp with time zone NOT NULL,
    create_dt timestamp with time zone NOT NULL DEFAULT now(),
    update_dt timestamp with time zone NOT NULL DEFAULT now(),
    ratio_value real,
    latest_record boolean NOT NULL DEFAULT false,
    universe_weight real,
    CONSTRAINT ratios_primarykey PRIMARY KEY (dtz, security_id, period_id),
    CONSTRAINT ratios_classification_id_fkey FOREIGN KEY (classification_id)
        REFERENCES settings.classification (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE NO ACTION,
    CONSTRAINT ratios_period_id_fkey FOREIGN KEY (period_id)
        REFERENCES settings.periods (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE NO ACTION,
    CONSTRAINT ratios_security_id_fkey FOREIGN KEY (security_id)
        REFERENCES settings.securitymaster (id) MATCH SIMPLE
        ON UPDATE NO ACTION
        ON DELETE NO ACTION
) PARTITION BY RANGE (dtz);

CREATE TABLE IF NOT EXISTS zpart.ratios_y1990 PARTITION OF research.ratios
    FOR VALUES FROM ('1990-01-01 00:00:00+00') TO ('1991-01-01 00:00:00+00')
    PARTITION BY LIST (period_id); 

CREATE TABLE IF NOT EXISTS zpart.ratios_y1991 PARTITION OF research.ratios
    FOR VALUES FROM ('1991-01-01 00:00:00+00') TO ('1992-01-01 00:00:00+00')
    PARTITION BY LIST (period_id);

等到今天

然后这些表中的每一个都由 period_id 分区

CREATE TABLE IF NOT EXISTS zpart.ratios_y1990p1 PARTITION OF zpart.ratios_y1990
    FOR VALUES IN ('1');
CREATE TABLE IF NOT EXISTS zpart.ratios_y1990p11 PARTITION OF zpart.ratios_y1990
    FOR VALUES IN ('11');

ETC

最后，请注意实际的表有一个主键：

CONSTRAINT ratios_primarykey PRIMARY KEY (dtz, security_id, period_id)

它适用于大多数用例。 但是，当我们通过 security_id 收集完整的历史记录时，需要很长时间，特别是一些查询让我认为我得到了错误的索引。

例如，有时我们会更新 security_id 的所有数据，并且我们希望清除任何可能被无效添加的旧数据。 下面是一个示例，其中我更新了每个有效分区中的 security_id，并且我想清除任何早期分区中的数据以防万一：

从 research.ratios 中删除 security_id=10450 和 dtz<'2017-01-01';

在这个例子中，有 189 个表要检查，它们都没有任何数据。 我原以为查询会检查 189 个表的每个主键索引，看到 security_id 10450 没有记录，然后完成。 然而，这个动作需要一分钟多的时间才能运行，这让我觉得我的索引不是我想象的那样。

这是解释分析的精简版：

QUERY PLAN
Delete on ratios  (cost=8118.25..4049804.56 rows=0 width=0) (actual time=67117.836..67117.938 rows=0 loops=1)
  Delete on ratios_y1990p1 ratios_1
  Delete on ratios_y1990p9 ratios_2
  ...
  Delete on ratios_y2016p15 ratios_188
  Delete on ratios_y2016p17 ratios_189
  ->  Append  (cost=8118.25..4049804.56 rows=48564 width=10) (actual time=67117.829..67117.931 rows=0 loops=1)
        ->  Bitmap Heap Scan on ratios_y1990p1 ratios_1  (cost=8118.25..9048.95 rows=268 width=10) (actual time=271.742..271.743 rows=0 loops=1)
              Recheck Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              ->  Bitmap Index Scan on ratios_y1990p1_pkey  (cost=0.00..8118.18 rows=268 width=0) (actual time=271.738..271.738 rows=0 loops=1)
                    Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Bitmap Heap Scan on ratios_y1990p9 ratios_2  (cost=6554.82..7340.83 rows=232 width=10) (actual time=9.389..9.390 rows=0 loops=1)
              Recheck Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              ->  Bitmap Index Scan on ratios_y1990p9_pkey  (cost=0.00..6554.76 rows=232 width=0) (actual time=9.384..9.384 rows=0 loops=1)
                    Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Bitmap Heap Scan on ratios_y1990p11 ratios_3  (cost=6907.45..7694.32 rows=232 width=10) (actual time=7.762..7.762 rows=0 loops=1)
              Recheck Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              ->  Bitmap Index Scan on ratios_y1990p11_pkey  (cost=0.00..6907.39 rows=232 width=0) (actual time=7.757..7.757 rows=0 loops=1)
                    Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Bitmap Heap Scan on ratios_y1990p12 ratios_4  (cost=7815.19..8617.27 rows=237 width=10) (actual time=6.257..6.258 rows=0 loops=1)
              Recheck Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              ->  Bitmap Index Scan on ratios_y1990p12_pkey  (cost=0.00..7815.13 rows=237 width=0) (actual time=6.253..6.253 rows=0 loops=1)
                    Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Seq Scan on ratios_y1990p14 ratios_5  (cost=0.00..7059.28 rows=257 width=10) (actual time=23.991..23.991 rows=0 loops=1)
              Filter: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              Rows Removed by Filter: 191152
        ->  Seq Scan on ratios_y1990p15 ratios_6  (cost=0.00..6500.28 rows=257 width=10) (actual time=23.207..23.208 rows=0 loops=1)
              Filter: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              Rows Removed by Filter: 191152
        ->  Seq Scan on ratios_y1990p17 ratios_7  (cost=0.00..7468.97 rows=234 width=10) (actual time=28.048..28.048 rows=0 loops=1)
              Filter: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
              Rows Removed by Filter: 191571

...

        ->  Index Scan using ratios_y2013p11_pkey on ratios_y2013p11 ratios_164  (cost=0.42..23840.35 rows=261 width=10) (actual time=53.111..53.111 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2013p12_pkey on ratios_y2013p12 ratios_165  (cost=0.42..25032.72 rows=265 width=10) (actual time=642.015..642.015 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2013p14_pkey on ratios_y2013p14 ratios_166  (cost=0.42..25227.62 rows=259 width=10) (actual time=87.942..87.943 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2013p15_pkey on ratios_y2013p15 ratios_167  (cost=0.42..24860.51 rows=259 width=10) (actual time=43.079..43.080 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2013p17_pkey on ratios_y2013p17 ratios_168  (cost=0.42..23706.46 rows=258 width=10) (actual time=77.406..77.407 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p1_pkey on ratios_y2014p1 ratios_169  (cost=0.42..25432.35 rows=258 width=10) (actual time=112.056..112.056 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p9_pkey on ratios_y2014p9 ratios_170  (cost=0.42..23804.65 rows=261 width=10) (actual time=98.315..98.315 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p11_pkey on ratios_y2014p11 ratios_171  (cost=0.42..23990.53 rows=261 width=10) (actual time=78.770..78.770 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p12_pkey on ratios_y2014p12 ratios_172  (cost=0.42..24980.20 rows=257 width=10) (actual time=298.219..298.219 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p14_pkey on ratios_y2014p14 ratios_173  (cost=0.42..25452.75 rows=257 width=10) (actual time=40.636..40.636 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p15_pkey on ratios_y2014p15 ratios_174  (cost=0.42..25024.75 rows=260 width=10) (actual time=791.547..791.548 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2014p17_pkey on ratios_y2014p17 ratios_175  (cost=0.42..23898.95 rows=266 width=10) (actual time=48.073..48.073 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p1_pkey on ratios_y2015p1 ratios_176  (cost=0.42..25477.92 rows=257 width=10) (actual time=53.338..53.339 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p9_pkey on ratios_y2015p9 ratios_177  (cost=0.42..23937.18 rows=260 width=10) (actual time=45.046..45.046 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p11_pkey on ratios_y2015p11 ratios_178  (cost=0.42..23910.06 rows=258 width=10) (actual time=46.683..46.683 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p12_pkey on ratios_y2015p12 ratios_179  (cost=0.42..25184.07 rows=259 width=10) (actual time=50.504..50.504 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p14_pkey on ratios_y2015p14 ratios_180  (cost=0.42..25492.90 rows=258 width=10) (actual time=49.115..49.115 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p15_pkey on ratios_y2015p15 ratios_181  (cost=0.42..25011.61 rows=259 width=10) (actual time=375.470..375.470 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2015p17_pkey on ratios_y2015p17 ratios_182  (cost=0.42..23847.07 rows=268 width=10) (actual time=44.908..44.908 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p1_pkey on ratios_y2016p1 ratios_183  (cost=0.42..25192.72 rows=258 width=10) (actual time=57.975..57.975 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p9_pkey on ratios_y2016p9 ratios_184  (cost=0.42..24203.92 rows=264 width=10) (actual time=47.379..47.379 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p11_pkey on ratios_y2016p11 ratios_185  (cost=0.42..23961.60 rows=260 width=10) (actual time=44.402..44.402 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p12_pkey on ratios_y2016p12 ratios_186  (cost=0.42..25132.85 rows=262 width=10) (actual time=50.321..50.321 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p14_pkey on ratios_y2016p14 ratios_187  (cost=0.42..25185.69 rows=260 width=10) (actual time=46.699..46.699 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p15_pkey on ratios_y2016p15 ratios_188  (cost=0.42..25009.06 rows=260 width=10) (actual time=377.000..377.000 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
        ->  Index Scan using ratios_y2016p17_pkey on ratios_y2016p17 ratios_189  (cost=0.42..23785.12 rows=262 width=10) (actual time=45.472..45.472 rows=0 loops=1)
              Index Cond: ((dtz < '2017-01-01 00:00:00+00'::timestamp with time zone) AND (security_id = 10450))
Planning Time: 66.162 ms
Execution Time: 67119.554 ms

只需确认：受上述影响的行数为零。

我猜我设置的表/索引不正确。 我正在寻找有关如何更好地管理分区/索引的建议，以便在没有 security_id 记录的情况下查询可以快速排除分区？ 请注意，有数十亿条记录，并且增长迅速。

Postgres 14.1

1 个解决方案

解决方案1
 2  2022-08-03 03:39:35

由于 security_id 不是 btree 索引中的第一列，它不能只跳转到所有给定 security_id 值所在的索引部分，因为没有这样的奇异位置。 您将需要另一个以 security_id 开头的索引，或者重新排列主键中的列顺序（追溯起来不是一件容易的事）。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 PostgreSQL日期函数扫描每个分区中的每一行，执行速度很慢 Postgres - 创建表非常慢 - 带有分区和 BRIN 索引 Postgres Inheritance 基于分区扫描所有分区 Postgresql查询非常慢 光滑的查询非常慢 Hibernate 启动很慢 带有 IN 的 postgres 查询非常慢 PostgreSQL非常慢 Select 明显很慢 对 GROUP BY 的查询非常慢
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM