Postgresql Table Partition
Question
I have a query regarding my query performance
I am partitioning a table on daily basis
The table creation script is mentioned below:
-- Table: myschema."auditrtailreference_2014-10-02"
-- DROP TABLE myschema."auditrtailreference_2014-10-02";
CREATE TABLE myschema."auditrtailreference_2014-10-02"
(
-- Inherited from table myschema.auditrtailreference: event smallint,
-- Inherited from table myschema.auditrtailreference: innodeid character varying(80),
-- Inherited from table myschema.auditrtailreference: innodename character varying(80),
-- Inherited from table myschema.auditrtailreference: sourceid character varying(300),
-- Inherited from table myschema.auditrtailreference: intime timestamp without time zone,
-- Inherited from table myschema.auditrtailreference: outnodeid character varying(80),
-- Inherited from table myschema.auditrtailreference: outnodename character varying(80),
-- Inherited from table myschema.auditrtailreference: destinationid character varying(300),
-- Inherited from table myschema.auditrtailreference: outtime timestamp without time zone,
-- Inherited from table myschema.auditrtailreference: bytes integer,
-- Inherited from table myschema.auditrtailreference: cdrs integer,
-- Inherited from table myschema.auditrtailreference: noofsubfilesinfile integer,
-- Inherited from table myschema.auditrtailreference: recordsequencenumberlist character varying(1000),
-- Inherited from table myschema.auditrtailreference: partial_cdrs integer,
-- Inherited from table myschema.auditrtailreference: duplicate_cdrs integer,
-- Inherited from table myschema.auditrtailreference: discarded_cdrs integer,
-- Inherited from table myschema.auditrtailreference: created_cdrs integer,
-- Inherited from table myschema.auditrtailreference: corrupted_cdrs integer,
-- Inherited from table myschema.auditrtailreference: created_files integer,
-- Inherited from table myschema.auditrtailreference: duplicate_files integer,
-- Inherited from table myschema.auditrtailreference: corrupted_files integer,
-- Inherited from table myschema.auditrtailreference: partial_files integer,
-- Inherited from table myschema.auditrtailreference: discarded_files integer,
-- Inherited from table myschema.auditrtailreference: empty_files integer,
CONSTRAINT "auditrtailreference_2014-10-02_intime_check" CHECK (intime >= '2014-10-02 00:00:00'::timestamp without time zone AND intime < '2014-10-03 00:00:00'::timestamp without time zone OR intime IS NULL),
CONSTRAINT "auditrtailreference_2014-10-02_outtime_check" CHECK (outtime >= '2014-10-02 00:00:00'::timestamp without time zone AND outtime < '2014-10-03 00:00:00'::timestamp without time zone OR outtime IS NULL)
)
INHERITS (myschema.auditrtailreference)
WITH (
OIDS=FALSE
);
ALTER TABLE myschema."auditrtailreference_2014-10-02"
OWNER TO erix;
-- Index: myschema."auditrtailreference_2014-10-02_dest_indx1"
-- DROP INDEX myschema."auditrtailreference_2014-10-02_dest_indx1";
CREATE INDEX "auditrtailreference_2014-10-02_dest_indx1"
ON myschema."auditrtailreference_2014-10-02"
USING btree
(destinationid COLLATE pg_catalog."default" );
-- Index: myschema."auditrtailreference_2014-10-02_in_indx1"
-- DROP INDEX myschema."auditrtailreference_2014-10-02_in_indx1";
CREATE INDEX "auditrtailreference_2014-10-02_in_indx1"
ON myschema."auditrtailreference_2014-10-02"
USING btree
(intime );
-- Index: myschema."auditrtailreference_2014-10-02_out_indx1"
-- DROP INDEX myschema."auditrtailreference_2014-10-02_out_indx1";
CREATE INDEX "auditrtailreference_2014-10-02_out_indx1"
ON myschema."auditrtailreference_2014-10-02"
USING btree
(outtime );
-- Index: myschema."auditrtailreference_2014-10-02_srce_indx1"
-- DROP INDEX myschema."auditrtailreference_2014-10-02_srce_indx1";
CREATE INDEX "auditrtailreference_2014-10-02_srce_indx1"
ON myschema."auditrtailreference_2014-10-02"
USING btree
(sourceid COLLATE pg_catalog."default" );
My Query for the data fetch is as follows
select t3.destinationid as input, t1.sourceid as Raw, t1.outtime::text, t7.destinationid, t7.outtime::text as output from myschema.auditrtailreference t1
LEFT JOIN myschema.auditrtailreference t2 on t2.sourceid = t1.destinationid AND t2.event ='80' and t2.outnodename not like '%CRS%' and t2.outnodename not like '%rch' and t2.outtime between '2014/12/11' AND '2014/12/12'
LEFT JOIN myschema.auditrtailreference t3 on t3.sourceid = t2.destinationid AND t3.event ='68' and t3.outtime between '2014/12/11' AND '2014/12/12'
LEFT JOIN myschema.auditrtailreference t4 on t4.sourceid = t3.destinationid AND t4.event ='67' and t4.innodename like 'AIR%ollect%r' and t4.outtime >= t3.outtime and t4.outtime between '2014/12/11' AND '2014/12/12'
LEFT JOIN myschema.auditrtailreference t5 on t5.sourceid = t4.destinationid AND t5.event ='80' and t5.outnodename not like '%ESB%' and t5.outnodename not like '%Type' and t5.outtime between '2014/12/11' AND '2014/12/12'
LEFT JOIN myschema.auditrtailreference t6 on t6.sourceid = t5.destinationid AND t6.event ='68' and t6.outtime between '2014/12/11' AND '2014/12/12'
LEFT JOIN myschema.auditrtailreference t7 on (t7.destinationid = t6.destinationid || '.gz' OR t7.destinationid = t6.destinationid OR t7.destinationid = t6.destinationid || '.csv') AND t7.event ='68' AND (t7.outnodename like '%AIR%_distributer' or t7.outnodename like '%AIR%_Arch' or t7.outnodename like '%AIR%_Distributer' or t7.outnodename like '%AIR%_Distributor' or t7.outnodename like '%AIR%_distributor') and t7.outtime between '2014/12/11' AND '2014/12/12'
where t1.event ='67' and t1.innodename like 'AIR%FTP' and t1.sourceid not like '%my%' and t1.intime >= '2014/12/11 00:00:00' and t1.intime <= '2014/12/11 23:59:59' AND t3.destinationid like '%';
I am facing the issue with the table performance Could someone please help me with this
Thanks a ton in Advance
Explain Analyze for the above query is as follows
Merge Right Join (cost=2230028197.24..10806551039995472.00 rows=432260982728698432 width=136) (actual time=597187.343..1059019.252 rows=3400 loops=1)
Merge Cond: ((t5.sourceid)::text = (t4.destinationid)::text)
-> Nested Loop Left Join (cost=0.03..24192368907.34 rows=14433968717 width=86) (actual time=679.865..1054884.883 rows=353487 loops=1)
Join Filter: (((t7.destinationid)::text = ((t6.destinationid)::text || '.gz'::text)) OR ((t7.destinationid)::text = (t6.destinationid)::text) OR ((t7.destinationid)::text = ((t6.destinationid)::text || '.csv'::text)))
-> Nested Loop Left Join (cost=0.03..15009299.32 rows=497219528 width=78) (actual time=184.474..13354.447 rows=353487 loops=1)
Join Filter: ((t6.sourceid)::text = (t5.destinationid)::text)
-> Merge Append (cost=0.03..7883009.19 rows=311384 width=78) (actual time=184.415..1877.546 rows=353487 loops=1)
Sort Key: t5.sourceid
-> Sort (cost=0.01..0.02 rows=1 width=336) (actual time=0.030..0.030 rows=0 loops=1)
Sort Key: t5.sourceid
Sort Method: quicksort Memory: 25kB
-> Seq Scan on auditrtailreference t5 (cost=0.00..0.00 rows=1 width=336) (actual time=0.001..0.001 rows=0 loops=1)
Filter: (((outnodename)::text !~~ '%ESB%'::text) AND ((outnodename)::text !~~ '%Type'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
-> Index Scan using "auditrtailreference_2014-12-11_srce_indx1" on "auditrtailreference_2014-12-11" t5 (cost=0.00..3842492.30 rows=311382 width=78) (actual time=99.449..896.976 rows=353478 loops=1)
Filter: (((outnodename)::text !~~ '%ESB%'::text) AND ((outnodename)::text !~~ '%Type'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
-> Index Scan using "auditrtailreference_2014-12-12_srce_indx1" on "auditrtailreference_2014-12-12" t5 (cost=0.00..4034803.06 rows=1 width=78) (actual time=84.927..699.589 rows=9 loops=1)
Filter: (((outnodename)::text !~~ '%ESB%'::text) AND ((outnodename)::text !~~ '%Type'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
-> Append (cost=0.00..22.85 rows=3 width=164) (actual time=0.006..0.025 rows=12 loops=353487)
-> Seq Scan on auditrtailreference t6 (cost=0.00..0.00 rows=1 width=336) (actual time=0.000..0.000 rows=0 loops=353487)
Filter: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 68::smallint))
-> Index Scan using "auditrtailreference_2014-12-11_srce_indx1" on "auditrtailreference_2014-12-11" t6 (cost=0.00..15.15 rows=1 width=78) (actual time=0.004..0.005 rows=1 loops=353487)
Index Cond: ((sourceid)::text = (t5.destinationid)::text)
Filter: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 68::smallint))
-> Index Scan using "auditrtailreference_2014-12-12_out_indx1" on "auditrtailreference_2014-12-12" t6 (cost=0.00..7.70 rows=1 width=78) (actual time=0.006..0.012 rows=11 loops=353486)
Index Cond: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone))
Filter: (event = 68::smallint)
-> Materialize (cost=0.00..60063.89 rows=1945 width=50) (actual time=0.000..0.982 rows=3400 loops=353487)
-> Append (cost=0.00..60054.16 rows=1945 width=50) (actual time=0.570..489.242 rows=3400 loops=1)
-> Seq Scan on auditrtailreference t7 (cost=0.00..0.00 rows=1 width=176) (actual time=0.001..0.001 rows=0 loops=1)
Filter: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 68::smallint) AND (((outnodename)::text ~~ '%AIR%_distributer'::text) OR ((outnodename)::text ~~ '%AIR%_Arch'::text) OR ((outnodename)::text ~~ '%AIR%_Distributer'::text) OR ((outnodename)::text ~~ '%AIR%_Distributor'::text) OR ((outnodename)::text ~~ '%AIR%_distributor'::text)))
-> Seq Scan on "auditrtailreference_2014-12-11" t7 (cost=0.00..60045.68 rows=1943 width=50) (actual time=0.568..487.333 rows=3399 loops=1)
Filter: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 68::smallint) AND (((outnodename)::text ~~ '%AIR%_distributer'::text) OR ((outnodename)::text ~~ '%AIR%_Arch'::text) OR ((outnodename)::text ~~ '%AIR%_Distributer'::text) OR ((outnodename)::text ~~ '%AIR%_Distributor'::text) OR ((outnodename)::text ~~ '%AIR%_distributor'::text)))
-> Index Scan using "auditrtailreference_2014-12-12_out_indx1" on "auditrtailreference_2014-12-12" t7 (cost=0.00..8.48 rows=1 width=50) (actual time=0.024..0.028 rows=1 loops=1)
Index Cond: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone))
Filter: ((event = 68::smallint) AND (((outnodename)::text ~~ '%AIR%_distributer'::text) OR ((outnodename)::text ~~ '%AIR%_Arch'::text) OR ((outnodename)::text ~~ '%AIR%_Distributer'::text) OR ((outnodename)::text ~~ '%AIR%_Distribu
tor'::text) OR ((outnodename)::text ~~ '%AIR%_distributor'::text)))
-> Materialize (cost=2230028197.20..2259975676.73 rows=5989495905 width=128) (actual time=3921.050..3924.851 rows=3400 loops=1)
-> Sort (cost=2230028197.20..2245001936.96 rows=5989495905 width=128) (actual time=3921.048..3922.459 rows=2576 loops=1)
Sort Key: t4.destinationid
Sort Method: quicksort Memory: 781kB
-> Merge Join (cost=591325.77..90441565.36 rows=5989495905 width=128) (actual time=3784.636..3918.059 rows=2576 loops=1)
Merge Cond: ((t3.sourceid)::text = (t2.destinationid)::text)
-> Sort (cost=308087.95..310241.46 rows=861404 width=120) (actual time=3464.384..3557.638 rows=150872 loops=1)
Sort Key: t3.sourceid
Sort Method: external merge Disk: 35784kB
-> Merge Left Join (cost=123584.91..170172.32 rows=861404 width=120) (actual time=2003.668..2499.298 rows=373330 loops=1)
Merge Cond: ((t3.destinationid)::text = (t4.sourceid)::text)
Join Filter: (t4.outtime >= t3.outtime)
-> Sort (cost=74053.15..74735.00 rows=272740 width=86) (actual time=1798.163..2042.079 rows=373330 loops=1)
Sort Key: t3.destinationid
Sort Method: external merge Disk: 39896kB
-> Append (cost=0.00..49428.59 rows=272740 width=86) (actual time=0.013..785.277 rows=373330 loops=1)
-> Seq Scan on auditrtailreference t3 (cost=0.00..0.00 rows=1 width=344) (actual time=0.001..0.001 rows=0 loops=1)
Filter: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND ((destinationid)::text ~~ '%'::text) AND (event = 68::smallint))
-> Seq Scan on "auditrtailreference_2014-12-11" t3 (cost=0.00..49420.12 rows=272738 width=86) (actual time=0.010..570.958 rows=373319 loops=1)
Filter: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND ((destinationid)::text ~~ '%'::text) AND (event = 68::smallint
))
-> Index Scan using "auditrtailreference_2014-12-12_out_indx1" on "auditrtailreference_2014-12-12" t3 (cost=0.00..8.47 rows=1 width=86) (actual time=0.021..0.030 rows=11 loops=1)
Index Cond: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone))
Filter: (((destinationid)::text ~~ '%'::text) AND (event = 68::smallint))
-> Sort (cost=49531.76..49536.49 rows=1895 width=86) (actual time=205.498..207.953 rows=7726 loops=1)
Sort Key: t4.sourceid
Sort Method: quicksort Memory: 459kB
-> Append (cost=0.00..49428.59 rows=1895 width=86) (actual time=0.251..202.613 rows=2576 loops=1)
-> Seq Scan on auditrtailreference t4 (cost=0.00..0.00 rows=1 width=344) (actual time=0.001..0.001 rows=0 loops=1)
Filter: (((innodename)::text ~~ 'AIR%ollect%r'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 67::smallint))
-> Seq Scan on "auditrtailreference_2014-12-11" t4 (cost=0.00..49420.12 rows=1893 width=86) (actual time=0.247..201.086 rows=2576 loops=1)
Filter: (((innodename)::text ~~ 'AIR%ollect%r'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 67::smallint))
-> Index Scan using "auditrtailreference_2014-12-12_out_indx1" on "auditrtailreference_2014-12-12" t4 (cost=0.00..8.47 rows=1 width=86) (actual time=0.022..0.022 rows=0 loops=1)
Index Cond: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone))
Filter: (((innodename)::text ~~ 'AIR%ollect%r'::text) AND (event = 67::smallint))
-> Materialize (cost=283237.82..290191.00 rows=1390636 width=86) (actual time=299.606..302.090 rows=2576 loops=1)
-> Sort (cost=283237.82..286714.41 rows=1390636 width=86) (actual time=299.543..300.435 rows=2576 loops=1)
Sort Key: t2.destinationid
Sort Method: quicksort Memory: 459kB
-> Nested Loop (cost=0.00..74796.60 rows=1390636 width=86) (actual time=0.251..296.114 rows=2576 loops=1)
Join Filter: ((t1.destinationid)::text = (t2.sourceid)::text)
-> Append (cost=0.00..52076.51 rows=940 width=86) (actual time=0.218..202.923 rows=2576 loops=1)
-> Seq Scan on auditrtailreference t1 (cost=0.00..0.00 rows=1 width=344) (actual time=0.001..0.001 rows=0 loops=1)
Filter: (((innodename)::text ~~ 'AIR%FTP'::text) AND ((sourceid)::text !~~ '%my%'::text) AND (intime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (intime <= '2014-12-11 23:59:59'::timestamp without time zone) AND (event = 67::smallint))
-> Seq Scan on "auditrtailreference_2014-12-11" t1 (cost=0.00..52076.51 rows=939 width=86) (actual time=0.216..201.425 rows=2576 loops=1)
Filter: (((innodename)::text ~~ 'AIR%FTP'::text) AND ((sourceid)::text !~~ '%my%'::text) AND (intime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (intime <= '2014-12-11 23:59:59'::timestamp without time zone) AND (event = 67::smallint))
-> Append (cost=0.00..24.13 rows=3 width=164) (actual time=0.007..0.030 rows=11 loops=2576)
-> Seq Scan on auditrtailreference t2 (cost=0.00..0.00 rows=1 width=336) (actual time=0.000..0.000 rows=0 loops=2576)
Filter: (((outnodename)::text !~~ '%CRS%'::text) AND ((outnodename)::text !~~ %rch'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
-> Index Scan using "auditrtailreference_2014-12-11_srce_indx1" on "auditrtailreference_2014-12-11" t2 (cost=0.00..15.99 rows=1 width=78) (actual time=0.005..0.006 rows=1 loops=2576)
Index Cond: ((sourceid)::text = (t1.destinationid)::text)
Filter: (((outnodename)::text !~~ '%CRS%'::text) AND ((outnodename)::text !~~ '%rch'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
-> Index Scan using "auditrtailreference_2014-12-12_out_indx1" on "auditrtailreference_2014-12-12" t2 (cost=0.00..8.14 rows=1 width=78) (actual time=0.004..0.016 rows=10 loops=2576)
Index Cond: ((outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone))
Filter: (((outnodename)::text !~~ '%CRS%'::text) AND ((outnodename)::text !~~ '%rch'::text) AND (event = 80::smallint))
Total runtime: 1059027.764 ms
(90 rows)
1 answers
solution1
0 2015-05-05 14:24:10
You should replace conditions:
outtime between '2014/12/11' AND '2014/12/12'
with:
outtime >= '2014-12-11 00:00:00'::timestamp without time zone AND outtime < '2014-12-12 00:00:00'::timestamp without time zone
Key difference is that between operator is both side inclusive whereas your partitions are defined as left side inclusive and right side exclusive.
This causes two scans for each myschema.auditrtailreference reference in your query, which is visible in plan as:
-> Index Scan using "auditrtailreference_2014-12-11_srce_indx1" on "auditrtailreference_2014-12-11" t5 (cost=0.00..3842492.30 rows=311382 width=78) (actual time=99.449..896.976 rows=353478 loops=1)
Filter: (((outnodename)::text !~~ '%ESB%'::text) AND ((outnodename)::text !~~ '%Type'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
-> Index Scan using "auditrtailreference_2014-12-12_srce_indx1" on "auditrtailreference_2014-12-12" t5 (cost=0.00..4034803.06 rows=1 width=78) (actual time=84.927..699.589 rows=9 loops=1)
Filter: (((outnodename)::text !~~ '%ESB%'::text) AND ((outnodename)::text !~~ '%Type'::text) AND (outtime >= '2014-12-11 00:00:00'::timestamp without time zone) AND (outtime <= '2014-12-12 00:00:00'::timestamp without time zone) AND (event = 80::smallint))
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Partition a table in mySQL
Oracle Partition Table : Data is not inserted in the right Partition
How to perform a filtered query in a PostgreSQL window partition?
Partition MySQL table by Column Value
How to partition a table in AWS Athena?
MySQL - clarification on table partition / subpartition
How to partition MySQL table by day?
Difficulty using CTE's with window PARTITION BY and string pattern matching in PostgreSQL
In-memory table in PostgreSQL
Index inside a table postgreSQL
Related Tags