Postgresql query is performing really bad

Question

I have a huge issue with query performance at my postgresql database. The version of postgresql is: "PostgreSQL 8.4.3 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20070115 (prerelease) (SUSE Linux), 64-bit"

I have config file set as follows:

shared_buffers = 8GB 
effective_cache_size = 24GB
work_mem = 419430kB 
maintenance_work_mem = 2GB 
checkpoint_segments = 128 
checkpoint_completion_target = 0.9 
wal_buffers = 16MB 
default_statistics_target = 500
constraint_exclusion = on

The query which I need to execute is:

SELECT events.evt_id, events.evt_time, events.device_evt_time , to_ip_char(events.sip) , evt_agent.port , events.rv40 , events.evt , events.msg , events.sun , events.rv35 , events.dun , events.rv45 , events.fn , events.dp , events.trgt_trust_name , events.trgt_trust_domain , events.rv36 , events.rv43 , events.cv21 , events.cv40 , events.cv41 , events.cv42 , events.cv43 , events.cv44 , events.cv50 , events.cv51 , events.cv52 , events.cv53 , events.cv54 , events.cv55 , events.cv56 , events.cv35 , events.cv60 , events.cv61 , events.cv62
FROM events, evt_agent
WHERE 
    events.agent_id = evt_agent.agent_id AND (evt_agent.port::text = ANY (ARRAY['x'::character varying, 'y'::character varying, 'z'::character varying]::text[])) 
    AND events.evt::text <> 'Internal Message'::text
    AND event_time > '2015-12-31 13:23:55.767+00'::timestamptz limit 10;
ORDER BY events.evt_time;

Table events is a partitioned table, one partition for each day. Each partition has two constrains:

Name    events_p_YYYYMMDDHHMISS_events_p_max_pk
Columns evt_time, evt_id
Name    events_p_YYYYMMDDHHMISS_dc  
Definition  evt_time > '2015-12-04 13:24:25.267973+00'::timestamp with time zone AND evt_time <= '2015-12-05 13:24:25.267973+00'::timestamp with time zone

and seven indexes:

Name    events_p_YYYYMMDDHHMISS_events_p_max_identity_ix1   
Columns evt_time, init_usr_identity_guid, rid02 
Operator classes    timestamptz_ops, uuid_ops, int8_ops

Name    events_p_YYYYMMDDHHMISS_events_p_max_identity_ix2   
Columns evt_time, trgt_usr_identity_guid, rid02 
Operator classes    timestamptz_ops, uuid_ops, int8_ops

Name    events_p_YYYYMMDDHHMISS_events_p_max_ix1    
Columns evt_time, sev, agent_id 
Operator classes    timestamptz_ops, int4_ops, int8_ops

Name    events_p_YYYYMMDDHHMISS_events_p_max_ix2    
Columns evt_time, dip, sev  
Operator classes    timestamptz_ops, int4_ops, int4_ops

Name    events_p_YYYYMMDDHHMISS_events_p_max_ix3    
Columns evt_time, res, sev  
Operator classes    timestamptz_ops, text_ops, int4_ops

Name    events_p_YYYYMMDDHHMISS_events_p_max_ix4    
Columns evt_time, sip, sev  
Operator classes    timestamptz_ops, int4_ops, int4_ops

Name    events_p_YYYYMMDDHHMISS_events_p_max_ix5    
Columns evt_time, txnmy_id, agent_id    
Operator classes    timestamptz_ops, int8_ops, int8_ops

Table evt_agent is a dictionary table with about 20-30 rows only.

constrains:

Name    evt_agent_pk    
Columns agent_id

indexes:

Name    evt_agent_ak1   
Columns agent, port, rn, pn, sn, st, device_ctgry, src_id, cust_id  
Operator classes    text_ops, text_ops, text_ops, text_ops, text_ops, text_ops, text_ops, uuid_ops, int8_ops

Name    evt_agent_ix1   
Columns device_ctgry, agent_id  
Operator classes    text_ops, int8_ops

When I execute the query I got this explain plan: http://explain.depesz.com/s/3xcu

Name    evt_agent_ix2   
Columns st, agent_id    
Operator classes    text_ops, int8_ops

Name    test_ev_ag_indx1    
Columns agent_id    
Operator classes    int8_ops

I thought there is a problem with statistics so I "vaccuum analyze" all related tables, but no improvement in the query performance, explain plan.

I tried with "inner query" trick, explain plan is here: http://explain.depesz.com/s/FAkH

From what I can understand it got kind of worse.

Do you have nay idea how to get better execution plan for this query? Now it takes about 42 minutes to get any results from the query.

Thanks in advance!

Answer 1

We will have to try and eliminate the hash join. A lateral join might just be the answer:

SELECT events.evt_id, events.evt_time, events.device_evt_time , to_ip_char(events.sip) , evt_agent.port , events.rv40 , events.evt , events.msg , events.sun , events.rv35 , events.dun , events.rv45 , events.fn , events.dp , events.trgt_trust_name , events.trgt_trust_domain , events.rv36 , events.rv43 , events.cv21 , events.cv40 , events.cv41 , events.cv42 , events.cv43 , events.cv44 , events.cv50 , events.cv51 , events.cv52 , events.cv53 , events.cv54 , events.cv55 , events.cv56 , events.cv35 , events.cv60 , events.cv61 , events.cv62
FROM evt_agent,  
  LATERAL (SELECT * 
           FROM events AS e
           WHERE
             e.agent_id = evt_agent.agent_id
             AND e.evt::text <> 'Internal Message'::text
             AND e.event_time > '2015-12-31 13:23:55.767+00'::timestamptz 
             LIMIT 10000) AS events
WHERE  (evt_agent.port::text = ANY (ARRAY['x'::character varying, 'y'::character varying, 'z'::character varying]::text[]))         
ORDER BY events.evt_time
LIMIT 10000;

Please try it with and without the inner limit and post the explain plans for both.