ORDER BY … ASC is slow and “Using index condition”

Question

I have 2 tables: user and post .

With show create table statements:

CREATE TABLE `user` (
  `user_id` bigint(20) NOT NULL AUTO_INCREMENT,
  `user_name` varchar(20) CHARACTER SET latin1 NOT NULL,
  `create_date` datetime DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`user_id`)
) ENGINE=InnoDB AUTO_INCREMENT=59 DEFAULT CHARSET=utf8;

CREATE TABLE `post` (
  `post_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `owner_id` bigint(20) NOT NULL,
  `data` varchar(300) CHARACTER SET latin1 DEFAULT NULL,
  PRIMARY KEY (`post_id`),
  KEY `my_fk` (`owner_id`),
  CONSTRAINT `my_fk` FOREIGN KEY (`owner_id`) REFERENCES `user` (`user_id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1012919 DEFAULT CHARSET=utf8;

Everything is fine util I execute 2 queries with ORDER BY statement and the result is very strange, ASC is slow but DESC is very fast.

SELECT sql_no_cache * FROM mydb.post where post_id > 900000 and owner_id = 20 order by post_id desc limit 10;
10 rows in set (0.00 sec)

SELECT sql_no_cache * FROM mydb.post where post_id > 900000 and owner_id = 20 order by post_id asc limit 10;
10 rows in set (0.15 sec)

Then I use explain statements:

explain SELECT sql_no_cache * FROM mydb.post where post_id > 900000 and owner_id = 20 order by post_id desc limit 10;
+----+-------------+-------+------+---------------+-------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys | key   | key_len | ref   | rows   | Extra       |
+----+-------------+-------+------+---------------+-------+---------+-------+--------+-------------+
|  1 | SIMPLE      | post  | ref  | PRIMARY,my_fk | my_fk | 8       | const | 239434 | Using where |
+----+-------------+-------+------+---------------+-------+---------+-------+--------+-------------+
1 row in set (0.01 sec)


explain SELECT sql_no_cache * FROM mydb.post where post_id > 900000 and owner_id = 20 order by post_id asc limit 10;
+----+-------------+-------+------+---------------+-------+---------+-------+--------+------------------------------------+
| id | select_type | table | type | possible_keys | key   | key_len | ref   | rows   | Extra                              |
+----+-------------+-------+------+---------------+-------+---------+-------+--------+------------------------------------+
|  1 | SIMPLE      | post  | ref  | PRIMARY,my_fk | my_fk | 8       | const | 239434 | Using index condition; Using where |
+----+-------------+-------+------+---------------+-------+---------+-------+--------+------------------------------------+
1 row in set (0.00 sec)

I think the point is Using index condition but I don't know why. How can I improve my database for better performance?

UPDATE:

explain SELECT * FROM mydb.post where post_id < 600000 and owner_id = 20 order by post_id desc limit 10;
+----+-------------+-------+------+---------------+-------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys | key   | key_len | ref   | rows   | Extra       |
+----+-------------+-------+------+---------------+-------+---------+-------+--------+-------------+
|  1 | SIMPLE      | post  | ref  | PRIMARY,my_fk | my_fk | 8       | const | 505440 | Using where |
+----+-------------+-------+------+---------------+-------+---------+-------+--------+-------------+


explain SELECT * FROM mydb.post where post_id < 600000 and owner_id > 19 and owner_id < 21 order by post_id desc limit 10;
+----+-------------+-------+-------+---------------+---------+---------+------+--------+-------------+
| id | select_type | table | type  | possible_keys | key     | key_len | ref  | rows   | Extra       |
+----+-------------+-------+-------+---------------+---------+---------+------+--------+-------------+
|  1 | SIMPLE      | post  | range | PRIMARY,my_fk | PRIMARY | 4       | NULL | 505440 | Using where |
+----+-------------+-------+-------+---------------+---------+---------+------+--------+-------------+

Answer 1

These are the relevant facts to understand this behavior:

• You are using InnoDB which uses a Clustered Index concept. The single interesting side effect of Clustered Indexes for your particular case is that every non-primary key index will also contain the primary key as the very last column in the index implicitly. No nedd for an index on (owner_id, post_id) — you already have it.

• MySQL can't resolve range conditions (<, >) on non-leading index columns in the right way. Instead, it will just ignore them during index lookup, and later on apply this part of the where clause as a filter. This is just a MySQL limitation to not start the scan directly at the position of post_id = 900000 — other databases do this very fine.

• When you are using DESC order, MySQL will start reading the index with the biggest post_id value it finds. It will then apply your filter post_id > 900000 . If it matches, it returns the row. Then it proceeds to the next row and so on until it has found 10 matching rows. However, all the matching rows are guaranteed to be where the index scan started.

• When you are using ASC order, MySQL starts reading the index at the other end, checks this value against post_id > 900000 and will probably need to discard the row because post_id is below that threshold. Now guess how many rows it needs to process this way before it finds the first row that matches post_id > 900000 ? That's what's eating up your time.

• "Using Index Condition" refers to Index Condition Pushdown: http://dev.mysql.com/doc/refman/5.6/en/index-condition-pushdown-optimization.html I'd say it should apply in both cases. However, it is not so relevant in the DESC case because the filter doesn't remove any rows anyway. In the ASC case it is very relevant and performance would be worst without it.

If you wan't to verify my statements, you could

• Increase/decrease the numeric value (900000) and see how the performance changes. Lower values should make ASC faster while keeping DESC fast too.

• Change the range condition > to < and see if it reverses the performance behavior of ASC / DESC . Remember that you might need to change the number to some lower value to actually see the performance difference.

How could one possibly know that?

http://use-the-index-luke.com/ is my guide that explains how indexes work.

Answer 2

It is nothing because “Using index condition” but how MySQL use INDEX and their query-engine works. MySQL use a simple query analyzer and optimizer.

In the case of post_id > 900000 and owner_id = 20 , you may notice it try to use key my_fk which is a "BIGGER INDEX" as it is sized in (64+32)*rows. It find all owner_id = 20 from index (yep, post_id was not used. stupid mysql)

After MySQL used a BIG and HEAVIER index to locate all the rows you need, it do another lookup to read actual rows (because you do SELECT * ) by their primary keys, (few more HDD seek here), and filter the result by using post_id > 900000 (SLOW)

In the case of order by post_id desc , it run faster could be many reason. One possible reason is the InnoDB cache, least inserted rows are warmer and easier to access then others.

In the case of post_id > 900000 and owner_id > 19 and owner_id < 20 , MySQL giveup the my_fk as a ranged scan on secondary index is not better then ranged scan on primary index.

It just use the PK to locate the right page of post_id 900000, and do a SEQUENCE READ from there, if your InnoDB page is not fragmented. (assume you are using AUTO_INCREMENT) scan some pages, and filter what match your need.

To do a "Optimization", (do it now) : Don't use SELECT *

To do a "Premature Optimization" (don't do it; don't do it yet); hint MySQL by USE INDEX ; create a index contains exact all the columns you need.

It is hard to say which is faster, my_fk and PK . Because the performance is various by the pattern of data. If owner_id = 20 is dominate or common in your table, using PK directly could be faster.

If owner_id = 20 is not common in your table, my_fk will give a boost as there are too many rows to read until (post_id > 900000 + XXX).

-- EDIT: BTW, try ORDER BY owner_id ASC, post_id ASC or DESC. MySQL will faster if it can just use the INDEX's order(not order the index).

Answer 3

I'm not a MySQL expert, but I don't think that either query is using an Index - unless there are indexes you have created which you haven't told us about. In thing that 'Using Index condition' is possibly an artefact of the way MySQL implements the LIMIT keyword.

If you put an index made up of (owner_id, post_id) on your post table it will help these two queries. In MySQL it should look something like:

create index ix_post_userpost on post (owner_id, post_id)

(I don't guarantee that syntax as I don't have MySQL.)