Mysql Query with Order By too slow

Question

I know this kind of question has been asked before, but none of those led me to a solution.

Here is my query:

SELECT p.photoid, c.comment, p.path, p.smallfile, p.bigfile, c.userid, c.dateposted, u.username, c.likephoto 
  FROM bs_photocomments c, photos p, user u 
  WHERE c.active = 1 
    AND u.userid = c.userid 
    AND p.photoid = c.photoid 
    AND c.id = (SELECT ID FROM bs_photocomments WHERE photoid = p.photoid ORDER BY ID DESC LIMIT 1) 
  ORDER BY c.id DESC LIMIT 2

A little explanation on the last AND :

Basically I am trying to get most recent comment for 20 unique photos. For example, if there are 5 comments in for one photo, I only want the last one to display.

The ORDER BY c.id is the killer. There are only 180k comments and 100k photos. If I remove that ORDER BY the query comes back instantly, but obviously not with the results I want.

Here are the indexes:

+------------------+------------+--------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table            | Non_unique | Key_name           | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+--------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| bs_photocomments |          0 | PRIMARY            |            1 | id          | A         |      183269 |     NULL | NULL   |      | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx1 |            1 | photoid     | A         |      183269 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx2 |            1 | active      | A         |           4 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx3 |            1 | dateposted  | A         |      183269 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx4 |            1 | userid      | A         |        4953 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx5 |            1 | puserid     | A         |        4072 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx6 |            1 | id          | A         |      183269 |     NULL | NULL   |      | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx6 |            2 | photoid     | A         |      183269 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx7 |            1 | photoid     | A         |      183269 |     NULL | NULL   | YES  | BTREE      |         |
| bs_photocomments |          1 | photocomments_idx7 |            2 | userid      | A         |      183269 |     NULL | NULL   | YES  | BTREE      |         |
+------------------+------------+--------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+


+--------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
| Table  | Non_unique | Key_name     | Seq_in_index | Column_name  | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
| photos |          0 | PRIMARY      |            1 | photoid      | A         |       57736 |     NULL | NULL   |      | BTREE      |         |
| photos |          1 | photos_idx1  |            1 | userid       | A         |        5773 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx2  |            1 | dateuploaded | A         |       57736 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx3  |            1 | camera       | A         |          17 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx4  |            1 | focallength  | A         |         308 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx5  |            1 | category     | A         |          96 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx6  |            1 | active       | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photo_idx7   |            1 | aperture     | A         |         354 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx8  |            1 | lenstype     | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx9  |            1 | film         | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |
| photos |          1 | photos_idx10 |            1 | originalfile | A         |       57736 |     NULL | NULL   | YES  | BTREE      |         |
+--------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+

Any ideas? Thanks!

Answer 1

This query will return the single most "recent" comment for (up to) 20 distinct photos (with "recent-cy" identified by ascending ID value on the bs_photocomments table). The 20 photos returned will be those that have the most "recent" comments, and be ordered from the most "recent" comment first. (NOTE: only rows in bs_photocomments that have active=1 are considered, all other rows in the bs_photocomments are ignored, as if they weren't even in the table.)

SELECT p.photoid
     , c.comment
     , p.path
     , p.smallfile
     , p.bigfile
     , c.userid
     , c.dateposted
     , u.username
     , c.likephoto
  JOIN ( 
         SELECT n.photoid
              , MAX(n.id) AS max_id
           FROM bs_photocomments n
          WHERE n.active = 1 
          GROUP BY n.photoid
          ORDER BY max_id DESC
          LIMIT 20
       ) m
  JOIN bs_photocomments c
    ON c.id = m.max_id
  JOIN photos p
    ON p.photoid = c.photoid
  JOIN user u
    ON u.userid = c.userid
 ORDER BY c.id DESC
 LIMIT 20

The "trick" is to use an inline view to get the "most recent" comment for each photoid (The inline view is aliased as m ). Once we have that, it's just a matter of getting the row from bs_photocomments, along with the related row from user and from photo.

I believe the real performance "killer" in the original query is the correlated subquery in the join predicate. That's going to be pretty expensive, in terms of executing that query for every row in the bs_photocomments table, or even running it for every row in the photos table.

---

For optimum performance, you'll want "covering indexes", rather than (useless) indexes on each separate column. At a minimum, you need an index on bs_photocomments that has photoid as the leading index, followed by id . Including the active column will mean that the inline view query can be satisfied "Using index", without need to visit any data pages.

... ON bs_photocomments (photoid, id, active)

With that index, MySQL should be able to get the maximum id for each photoid fairly efficiently. Likewise, having an index

... ON bs_photocomments (id, comment, userid, dateposted, likephoto)

will mean that the outer query against the bs_photocomments table ( c ) can be satisfied "Using index" as well. (MySQL may be able to satisfy the ORDER BY from the index, without a need for a sort operation. (You'd need to check the EXPLAIN output to see if "Using filesort" appears in the Extra column.)

Also, these covering indexes may slightly improve the performance of the query. (The EXPLAIN output will show "Using index" if the query is satisfied from the index without accessing pages in the underlying table.)

  ON photos (photoid, path, smallfile, bigfile)

  ON user (userid, username)

Most of those separate indexes on each individual column are useless; it's very unlikely that any of those indexes would ever be used, so all they do is consume resources.