Painfull slow MySQL query when using ORDER BY (And indexed)

Question

I have been researching but unable to solve this slow ORDER by clause on aa fairly large table (500k rows).

I have the following query:

SELECT gifts.gift_id, gifts.gift_title, gifts.gift_price, gifts.gift_image, gifts.gift_slug
FROM gifts
LEFT JOIN tags_gifts_occasion_specific AS os ON gifts.gift_id = os.gift_id
LEFT JOIN popularity ON popularity.gift_id = gifts.gift_id
WHERE published = '1'
AND (
os.tag_id IS NULL
)
ORDER BY popularity.popularity DESC , gift_id DESC

It works fast (0.0007 seconds), until the ORDER BY popularity.popularity is added, and then it takes 4.7 seconds!

The popularity table does have indexes although I understand this is irrelevant for sorting. I just don;t understand why this is running so slowly. When I had popularity in the same table it was fast, but I have now moved it into a separate table for performance and for increased functionality.

Any advice for this greatly appreciated.

EXPLAIN:

1   SIMPLE  gifts   ref     index_published     index_published     1   const   494384  Using where; Using temporary; Using filesort
1   SIMPLE  os  ref     Gift ID     Gift ID     4   gifts.gift_id   1   Using where; Using index; Not exists
1   SIMPLE  popularity  ref     Unique,Index Gift ID    Unique  4   gifts.gift_id   1   

POPULARITY TABLE INDEXES:

The table has 3 columns all of which their own index (gift_id, tag_id, popularity) Tag ID is not used in this search There is also a UNIQUE index for the three columns

Edit    Drop Drop   Unique  BTREE   Yes No  gift_id 26019   A   No  
tag_id  26019   A   No
Edit Edit   Drop Drop   Index Gift ID   BTREE   No  No  gift_id 26019   A   No  
Edit Edit   Drop Drop   Index Tag ID    BTREE   No  No  tag_id  3   A   No  
Edit Edit   Drop Drop   Index Popularity    BTREE   No  No  popularity  351 A   No  

Answer 1

I am curious how this version of the query performs when you have the index popularity(popularity desc, gift_id) :

select g.gift_id, g.gift_title, g.gift_price, g.gift_image, g.gift_slug, p.popularity
from popularity p join
     gifts g
     on p.gift_id = g.gift_id
where g.published = '1' and
      not exists (select 1
                  from tags_gifts_occasion_specific tgos
                  where tgos.gift_id = p.gift_id
                 )
order by p.popularity DESC, g.gift_id DESC;

This isn't exactly the same as your query, because gifts with no popularity are not included in the result set. This version might convince MySQL to use the above index on popularity , with the rest of the work being done with index lookups.

EDIT:

If you are willing to run this as two queries, you can do the above query and then:

select g.gift_id, g.gift_title, g.gift_price, g.gift_image, g.gift_slug
from gifts g
where g.published = '1' and
      not exists (select 1
                  from tags_gifts_occasion_specific tgos
                  where tgos.gift_id = p.gift_id
                 ) and
      not exists (select 1
                  from popularity p
                  where p.gift_id = g.gift_i
order by g.gift_id DESC;

You could even combine these with union all . I sort of discourage that. It would probably work in practice, but there is no explicit guarantee that the results of the union all come from the first query followed by the second.