mysql index seems to fail in large table

Question

I use a large (>100 million rows) table with the following create:

 data_values | CREATE TABLE `data_values` (
  `T` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `ID` varchar(32) NOT NULL,
  `VAL` float DEFAULT NULL,
  PRIMARY KEY (`T`,`ID`),
  KEY `IX_d` (`ID`),
  KEY `IX_T` (`T`)
) ENGINE=InnoDB DEFAULT CHARSET=greek PACK_KEYS=1 ROW_FORMAT=DYNAMIC |

My queries are of the following form:

select t,id,val from data_values where t between "2013-01-11 02:47:02" and "2013-01-12 04:59:02" and id="815";

My problem is that sometimes the index is used (and the answer comes pretty fast), and sometimes it is not used (and the answer comes in 1-2 minutes), and I can't find any logic behind that. For example the following query is fast:

select t,id,val from data_values where t between "2013-01-11 02:47:02" and "2013-01-13 04:59:02" and id="815";

while the following query (which asks for less data, one day instead of two) is slow:

select t,id,val from data_values where t between "2013-01-11 02:47:02" and "2013-01-12 04:59:02" and id="815";

Explaining the queries:

explain select t,id,value from data_values where t between "2013-01-11 02:47:02" and "2013-01-13 04:59:02" and id="815";
| id | select_type | table                | type        | possible_keys
| key             | key_len | ref  | rows | Extra
          |
+----+-------------+----------------------+-------------+-----------------------
+-----------------+---------+------+------+-------------------------------------
----------+
|  1 | SIMPLE      | data_values           | index_merge | PRIMARY,IX_D,IX_T
| IX_D,PRIMARY     | 34,38  | NULL | 1033 | Using intersect(IX_D,PRIMARY); Us
ing where |
+----+-------------+----------------------+-------------+-----------------------
+-----------------+---------+------+------+-------------------------------------


explain select t,id,val from data_values where t between "2013-01-11 02:47:02" and "2013-01-12 04:59:02" and id="815";
+----+-------------+----------------------+------+-----------------------+------
---+---------+-------+--------+-------------+
| id | select_type | table                | type | possible_keys         | key
   | key_len | ref   | rows   | Extra       |
+----+-------------+----------------------+------+-----------------------+------
---+---------+-------+--------+-------------+
|  1 | SIMPLE      | data_values          | ref  | PRIMARY,IX_D,IX_T     | IX_D            | 34      | const | 143900 | Using where |
+----+-------------+----------------------+------+-----------------------+------
---+---------+-------+--------+-------------+

I don't understand how the first query, which is "broader" (asks for more data) uses the index and the second one does not. Searching stackoverflow I have found some answers regarding multiple-column indexes. But I already use one (my primary key),and in the correct order (in the queries, t variable is mentioned before the id). Another answer I have found concerns queries with OR (do separate queries and form a UNION) but my query uses AND.

I need advice on how I can speed up my queries,

thanks a lot.

Answer 1

You have an index on (t, id) .

You need an index on (id, t) for this kind of query, or even better, a "covering" index on (id, t, val) :

ALTER TABLE data_values
    DROP INDEX IX_T           -- the Primary Key is (t, id) so this index 
                              -- is rather redundant.
  , DROP INDEX IX_d           -- this was not redundant but it is now
                              -- (because of) 
  , ADD INDEX IX_ID_T_VAL     -- the new index
        (id, t, value) ;