Adding LIMIT to query increases query time by over 1000%

Question

I am running the following query:

SELECT 
        MyField,
        COUNT(*) AS MyCount
    FROM
        MyTable
    NATURAL JOIN
        AnotherTable
    WHERE
        Timestamp >= 1000 AND Timestamp <= 10000
    GROUP BY
        MyField
    ORDER BY
        MyCount DESC;

This runs fine and takes about 6 seconds to complete. If I want to limit the result to show only the 20 highest MyCount s, I add LIMIT 20 on to the end of the query. Suddenly it takes 6 minutes to complete!

The EXPLAIN output for the original query:

+----+-------------+-------------+--------+---------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table       | type   | possible_keys             | key     | key_len | ref                       | rows    | Extra                                        |
+----+-------------+-------------+--------+---------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
|  1 | SIMPLE      | MyTable     | ALL    | mytable_fkey              | NULL    | NULL    | NULL                      | 6858209 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | AnotherTable| eq_ref | PRIMARY                   | PRIMARY | 4       | test.MyTable.FKeyID       |       1 | Using index                                  |
+----+-------------+-------------+--------+---------------------------+---------+---------+---------------------------+---------+----------------------------------------------+

The EXPLAIN output for the query with LIMIT 20 :

+----+-------------+-------------+--------+---------------------------+-------------------------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table       | type   | possible_keys             | key                     | key_len | ref                       | rows    | Extra                                        |
+----+-------------+-------------+--------+---------------------------+-------------------------+---------+---------------------------+---------+----------------------------------------------+
|  1 | SIMPLE      | MyTable     | index  | mytable_fkey              | myfield_timestamp_index | 771     | NULL                      | 6858209 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | AnotherTable| eq_ref | PRIMARY                   | PRIMARY                 | 4       | test.MyTable.FKeyID       |       1 | Using index                                  |
+----+-------------+-------------+--------+---------------------------+-------------------------+---------+---------------------------+---------+----------------------------------------------+

What is the explanation for this? Is there a better way I can limit the number of rows?

Answer 1

If you see Using temporary; Using filesort Using temporary; Using filesort in your EXPLAIN output, you are probably missing a suitable index and you're getting killed because of it.

Make sure your JOIN and GROUP BY fields are both available in the same index.