Simple MySQL select query takes 4 hours

Question

I'd be grateful if you could help with a novice question. I'm apply the following SQL:

INSERT INTO t03_hesid_history(uniqueID, hes_data_all_years.extract_hesid, FIELD1, FIELD2)  
SELECT uniqueID, hes_data_all_years.extract_hesid, FIELD1, FIELD2  
FROM hes_data_all_years  
INNER JOIN T02_hesid_grouped  
ON hes_data_all_years.extract_hesid = T02_hesid_grouped.extract_hesid;

The hes_data_all_years table has 188 million records and the T02_hesid_grouped table has 80,000 records. The T02_hesid_grouped table has a single (unique) field called extract_hesid which is indexed. The hes_data_all_years has many fields and a single index on the extract_hesid field that is being joined.

The query aims to extract all records in hes_data_all_years with a match in the T02_hesid_grouped field. I expect the output to provide 1-2m records.

The query takes approximately 4 hours...

Is the length of time due to the dataset size or is there some optimization that could be carried out? Many thanks!!

EXPLAIN outputon the SELECT part is shown below:

1   SIMPLE  T02_hesid_grouped   index   I_HESID I_HESID 43      79824   Using index
1   SIMPLE  hes_data_all_years  ref I_HESID I_HESID 43  hes.T02_hesid_grouped.extract_hesid 1   Using where

Answer 1

This could be a performance problem with generating the resultset or with inserting it into the destination table.

Ordinarily one doesn't do SELECT * for a resultset that's being used for an insert, but rather names the columns to select in the same order as the fields into the destination table. Your resultset has two columns named extract_hesid . It seems unlikely that's what you want.

What is the value of hes_data_all_years.extract_hesid for the rows in hes_data_all_years that don't match rows in T02_hesid_grouped ? Things will be faster if those values aren't NULL.

Are your tables, especially the destination table, using MyISAM? Things will be faster if they are because InnoDB is transaction oriented, and has to generate rollback data while it's doing that INSERT of a couple of megarows.

188 megarows isn't small, and your elapsed time isn't totally outrageous. It is long, but not absurdly so. You may want to check that your MySQL server has enough RAM. Or, if this is a yearly or one time thing, you may want to simply declare victory and move on.