Postgres 9.1 on Amazon EC2 m1.large sometimes has a really slow query

Question

I have seen this query for various personid's and different types take less than 10 ms and I also seen it take as much 16min. What is going on?

    Column    |       Type       |                       Modifiers
--------------+------------------+--------------------------------------------------------
 id           | bigint           | not null default nextval('record_seq'::regclass)
 type         | integer          | not null
 personid     | integer          |
 reporttime   | bigint           | not null
 totalreading | double precision | not null
 delta        | double precision | not null

Indexes:
    "record_pkey" PRIMARY KEY, btree (id)
    "record_personid_idx" btree (personid)
    "record_type_idx" btree (type)
    "record_reporttime_idx" btree (reporttime) CLUSTER

This is an explain analyze of the query that is sometimes really slow.

explain analyze SELECT ID, TYPE, REPORTTIME, TOTALREADING, DELTA, PERSONID FROM RECORD WHERE PERSONID=1103 AND TYPE=405 AND REPORTTIME <= 1332447354000 ORDER BY REPORTTIME DESC LIMIT 1;

 Limit  (cost=0.00..327.93 rows=1 width=52) (actual time=239749.274..239749.274 rows=0 loops=1)
   ->  Index Scan Backward using record_reporttime_idx on record  (cost=0.00..1196290.82 rows=3648 width=52) (actual time=239749.251..239749.251 rows=0 loops=1)
         Index Cond: (reporttime <= 1332447354000::bigint)
         Filter: ((personid = 1103) AND (type = 405))
 Total runtime: 239749.409 ms

In there are roughly 10-20 types, with 2 being most heavily used, 405 isn't used nearly as often.

 select count(*) from record;
  count
----------
 30420232

 SELECT COUNT(*) FROM record WHERE PERSONID=1103 AND TYPE=405;
  count
-------
    58

Answer 1

Because you're searching for the last reporttime value before some target the planner thinks searching backwards on that makes sense.

It probably does some/most of the time but occasionally you're having to go a long way back to find the right combination of (personid,type).

If you generally search on (personid,type) try a combined index on both, or perhaps even all three columns. The ordering of the three columns and whether you need to keep the other indexes will depend on your total mix of queries.