MySQL - indexing - what am I doing wrong?

Question

I have a DB with ~400 tables. In each are 5 columns with anywhere from 1M-200M rows. One of the columns is a DATE value and is one index. (There is also a different unique key).

When walking through these tables by doing "select * from table where <date field> = ?" I find that the response time is proportional to where the date comes row-wise. IE If the earliest date is "2001-01-01", the most recent date is "2010-12-01" and there are roughly an equal number of rows on each date that select is much slower for 2010 than 2001.

The data is written to the table in increasing date order (lower PK values = earlier dates).

Doing a 'desc' on the select shows that it's using the appropriate index and getting the appropriate number of values.

Is this expected behavior?

Answer 1

Have you tried running an ANALYZE or OPTIMIZE TABLE on them? Maybe the indexes need restructuring.

Something I haven't tried before but have seen as a solution, is to split your date into 3 separate fields: thisYear YEAR,
thisMonth TINYINT,
thisDate TINYINT

and index:
INDEX dateIDX (thisYear, thisMonth, thisDate)

and search on those:
WHERE thisYear = 2001 AND thisMonth = 1 AND thisDate = 1

If you're using MySQL 5.1 or later, you can try partitioning your data. http://dev.mysql.com/doc/refman/5.1/en/partitioning-range.html For previous versions, MERGE tables may help too.