Slow MySQL SELECT query

Question

I've got a slow MySQL SELECT query, that I can't seem to troubleshoot.

It's a simple one, on a table with about 600,000 records.

SELECT * 
FROM  `civicrm_contact` contact
WHERE contact.external_identifier =123456

The Select query takes anywhere between 3-6 seconds, which make importing another 600,000 records that depend on this query, completely impractical.

The table indexes are shown in attached image:

If I search based on contact.id=123456 then the query time is down to about 0.004s. contact.id is the primary key on the table. external_identifier is a unique index.

Answer 1

I know this is an old thread but as it relates to CiviCRM I thought I'd push out my thoughts. The fix is actually not best practise as you've altered one of the core packaged tables to get your query running faster. Although this may be ok for you I definitely would not recommend this to everyone.

Your solution has probably highlighted the problem with the query though, it seems you're telling the query that your expecting a number but in actually fact the data is stored as a VARCHAR. So I think simply putting single quotes around the value would have done the trick?

SELECT * FROM civicrm_contact contact WHERE contact.external_identifier = '123456'

Without this I'm pretty sure (having worked with Oracle for a number of years) that an implicit data type conversion would take place therefore the query is then not able to use the INDEX.

An explain plan should prove this theory.

Thanks

Parvez

Answer 2

It seems you use a BTREE index. If you don't perform any range queries on this column (using < , > , <= or >= ), you may want to use a hash based index.

See Comparison of B-Tree and Hash Indexes for detailed information.

And see here exact syntax.

Answer 3

I changed the structure to make external_identifier of type INT instead of VARCHAR. Speed has increased to 0.006s

I'm not sure yet whether or not this will have any broader implications

Answer 4

I'm dubious of the data type conversion being the issue. I wonder if it's about the size limit of the indexed field. Being a btree means the index keys are potentially much bigger than a hash key would be. Is this necessary? Would it be better to store the external ids in a separate table, and link them based on a numeric id?

More questions than answers here really.