Really slow performance in postgres

Question

I have a table where i only insert rows, never delete. On each loop i insert around 36k rows.

I need to get rows from this table to perfom operations. The problem is on every loop the query perfomance is really really poor.

For example, on the loop 31:

explain analyze select exp(least(709,a.value)), a.from, a.to,a.material,a.transport from resultTable a where a.loop=31;
                                                                           QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on resultTable a  (cost=36.58..4431.79 rows=2425 width=48) (actual time=7.351..33894.217 rows=34640 loops=1)
   Recheck Cond: (loop = 31)
   ->  Bitmap Index Scan on "resultTable_idx_mo"  (cost=0.00..35.97 rows=2425 width=0) (actual time=4.880..4.880 rows=34640 loops=1)
         Index Cond: (loop = 31)
 Total runtime: 33897.070 ms
(5 rows)

For the loop 43:

explain analyze select exp(least(709,a.value)), a.from, a.to,a.material,a.transport from resultTable a where a.loop=43;
                                                                           QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on resultTable a  (cost=36.58..4431.79 rows=2425 width=48) (actual time=10.129..125460.445 rows=34640 loops=1)
   Recheck Cond: (loop = 43)
   ->  Bitmap Index Scan on "resultTable_idx_mo"  (cost=0.00..35.97 rows=2425 width=0) (actual time=4.618..4.618 rows=34640 loops=1)
         Index Cond: (loop 43)
 Total runtime: 125463.516 ms
(5 rows)

The time is growing as exponential. I am doing VACUUM and REINDEX in every loop (I have tried also without, but results are same).

Any idea how to improve the time?

Thanks in advance.

After partition:

    QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Result  (cost=14.47..2686.29 rows=1649 width=48) (actual time=18.562..220124.597 rows=34640 loops=1)
   ->  Append  (cost=14.47..2682.17 rows=1649 width=48) (actual time=5.189..32.743 rows=34640 loops=1)
         ->  Bitmap Heap Scan on resultTable a  (cost=14.47..1655.44 rows=866 width=48) (actual time=0.008..0.008 rows=0 loops=1)
               Recheck Cond: (loop = 60)
               ->  Bitmap Index Scan on "resultTable_idx_mo"  (cost=0.00..14.26 rows=866 width=0) (actual time=0.006..0.006 rows=0 loops=1)
                     Index Cond: (loop = 60)
         ->  Bitmap Heap Scan on result_table_child_70 a  (cost=8.82..1026.73 rows=783 width=48) (actual time=5.181..29.068 rows=34640 loops=1)
               Recheck Cond: (loop = 60)
               ->  Bitmap Index Scan on resultTable_child_70_idx (cost=0.00..8.63 rows=783 width=0) (actual time=4.843..4.843 rows=34640 loops=1)
                     Index Cond: (loop = 60)
 Total runtime: 220128.290 ms

After analyze the table and setting enable_bitmapscan=off (still using partitioning):

 Result  (cost=0.00..2761.06 rows=33652 width=389) (actual time=9.714..378389.177 rows=34640 loops=1)
   ->  Append  (cost=0.00..2676.93 rows=33652 width=389) (actual time=0.119..34.065 rows=34640 loops=1)
         ->  Index Scan using "resultTable_idx_mo" on resultTable a  (cost=0.00..12.84 rows=5 width=48) (actual time=0.058..0.058 rows=0 loops=1)
               Index Cond: (loop= 79)
         ->  Index Scan using resultTable_child_80_idx on resultTable_child_80 a  (cost=0.00..2664.10 rows=33647 width=389) (actual time=0.061..30.303 rows=34640 loops=1)
               Index Cond: (loop = 79)
 Total runtime: 378393.671 ms
(7 rows)

Answer 1

If clustering and partitioning isn't working, I start to suspect you have some serious issue with your storage system. 10s of seconds to heap scan 35K rows is several orders of magnitude too slow. What version of Postgres are you using? What does your storage look like? Check your iostats.

I set up an experiment on a smallish VM (1 fractional cpu, 1GB mem, NFS disk mount) with Pg 9.0.4 making your table and index and injecting 1000 batches of 36000 records each.

insert into r(loop,value,xfrom,xto,material,transport,pk) select 0,0,0,0,0,0,i from generate_series(0,35999) i;
insert into r(loop,value,xfrom,xto,material,transport,pk) select 1,0,0,0,0,0,i from generate_series(36000,71999) i;
...

Running your select on any batch is consistently under 40ms:

explain analyze select exp(least(709,a.value)), a.xfrom, a.xto, a.material,a.transport from r a where a.loop=31;
 Index Scan using "resultTable_idx_mo" on r a  (cost=0.00..1596.35 rows=37680 width=21) (actual time=0.087..34.038 rows=36000 loops=1)
   Index Cond: (loop = 31)
 Total runtime: 36.332 ms

explain analyze select exp(least(709,a.value)), a.xfrom, a.xto,a.material,a.transport from r a where a.loop=87;
 Index Scan using "resultTable_idx_mo" on r a  (cost=0.00..1421.35 rows=33480 width=21) (actual time=0.105..37.357 rows=36000 loops=1)
   Index Cond: (loop = 87)
 Total runtime: 39.365 ms

Note my plans come up with ordinary IndexScan's instead of BitmapScans followed by HeapScans. Have you tweaked with plan optimization config to influence the plan?

In your explains, I note the estimated row count is far less than the actual row count (1649 vs 34640). This indicates you have inaccurate statistics on the table. You should run ANAYLZE resultTable;

I believe there's something badly busted in your postgres config or your storage system when these simple index scan selects take more than a few tens of ms. Is there other activity on this db when you run your queries? Perhaps your results are competition for resources with other queries?

Answer 2

Cluster will probably speed it up:

cluster resultTable using resultTable_idx_mo;

But the definitive solution would be to partition your table by loop:

http://www.postgresql.org/docs/current/interactive/ddl-partitioning.html