Joining two tables MySQL is slow even with indexing

Question

I have two tables:

The first one table1 is created on the fly (as the user submits data from a web server) and is usually ~50K rows. The second table is a lookup table table2 and has ~10Mil rows.

I'm trying to join both tables on four columns as follows:

SELECT t.id FROM table1 t 
JOIN table2 m
ON (t.name = m.name AND t.pos = m.pos AND t.ref = m.ref AND t.alt = m.alt);

I've indexed columns name (VARCHAR), pos (INT), ref (CHAR) and alt (CHAR) in table2 , but the query still takes way too long to complete.

Any pointers on what could be going wrong here?

Thanks

---

Output of EXPLAIN :

id  select_type table   partitions  type    possible_keys   key key_len ref rows    filtered    Extra
1   SIMPLE  t1  NULL    ALL NULL    NULL    NULL    NULL    49329   100.00  Using where
1   SIMPLE  t2  NULL    ref table2_name,table2_pos,table2_ref,table2_alt    table2_name 32  my_db.t1.NAME   2488    0.00    Using index condition; Using where

Answer 1

Create a compound index on name, pos, ref, alt

like

INDEX theIndex (name,pos,ref, alt)

Also, 4 single indices will help a little bit - see http://dev.mysql.com/doc/refman/5.7/en/index-merge-optimization.html - but not as much as a compound index.

Answer 2

Two things to try here:

1. The first is the easiest -- change the order of your join clause to move the varchar column to the end.

SELECT t.id FROM table1 t JOIN table2 m ON (t.pos = m.pos AND t.ref = m.ref AND t.alt = m.alt AND t.name = m.name);

2. This is a little more work, but add a new computed column that generates a numeric hash based on the 4 columns. Remove the indexes on pos, ref, alt and name and add a new index to the the hash column. Then include the hash column in your join clause.

SELECT t.id FROM table1 t JOIN table2 m ON (t.hash = m.hash AND t.pos = m.pos AND t.ref = m.ref AND t.alt = m.alt AND t.name = m.name);

Edit: Without looking at your database and the query execution plan, it's hard to troubleshoot this, but my guess is that MySQL is having a hard time joining on the VARCHAR column. Can you update your question with the results of

EXPLAIN SELECT t.id FROM table1 t JOIN table2 m ON (t.name = m.name AND t.pos = m.pos AND t.ref = m.ref AND t.alt = m.alt)

Answer 3

Index for any tables:

SELECT 
  CONCAT(
    'ALTER TABLE ', 
    TABLE_NAME, 
    ' ', 
    'ADD ', 
    IF(
      NON_UNIQUE = 1, 
      CASE UPPER(INDEX_TYPE) WHEN 'FULLTEXT' THEN 'FULLTEXT INDEX' WHEN 'SPATIAL' THEN 'SPATIAL INDEX' ELSE CONCAT(
        'INDEX ', INDEX_NAME, ' USING ', INDEX_TYPE
      ) END, 
      IF(
        UPPER(INDEX_NAME) = 'PRIMARY', 
        CONCAT(
          'PRIMARY KEY USING ', INDEX_TYPE
        ), 
        CONCAT(
          'UNIQUE INDEX ', INDEX_NAME, ' USING ', 
          INDEX_TYPE
        )
      )
    ), 
    '(', 
    GROUP_CONCAT(
      DISTINCT CONCAT('', COLUMN_NAME, '') 
      ORDER BY 
        SEQ_IN_INDEX ASC SEPARATOR ', '
    ), 
    ');'
  ) AS 'Show_Add_Indexes' 
FROM 
  information_schema.STATISTICS 
WHERE 
  TABLE_SCHEMA = 'your_database' 
  and TABLE_NAME = 'your_table';
-- GROUP BY 
--   TABLE_NAME, 
--   INDEX_NAME 
-- ORDER BY 
--   TABLE_NAME ASC, 
--   INDEX_NAME ASC;