Faster search query for IN statement with SELECT in MySQL

Question

I'm currently doing some query for my app and I need to get the nearest store on my current position and to do this first I need to get all the item that has the same name then get it's information and trim down that query. Now I used IN statement for this but since the items being searched are also based on a list I need to make use of another select for this here is my code so far:

select *
from product p,
store s,
branches b
where 1 = 1
and b.idproduct = p.idproduct
and p.store = s.idstore
and common_name IN(SELECT p.common_name
FROM shopping_list_content s, product p
WHERE 1 =1
AND s.iditem = p.idproduct
AND s.idlist =$listid)

Now it works as I wanted it to be but I wanted it to do the query faster than this. For now it takes more than 3 seconds for this query to run faster than this. much better if it is less than a second. Any other option I can use for this?

Answer 1

MySQL has difficulty optimising subqueries, when you write something like:

SELECT  *
FROM    T
WHERE   T.ID (SELECT ID FROM T2);

It is sometimes rewritten as

SELECT  *
FROM    T
WHERE   EXISTS
        (   SELECT  1
            FROM    T2
            WHERE   T.ID = T2.ID
        );

The subquery is then executed once per row in T , whereas if you write:

SELECT  T.*
FROM    T
        INNER JOIN
        (   SELECT  DISTINCT ID
            FROM    T2
        ) T2
            ON T2.ID = T.ID;

Your result set will be the same, but MySQL will first fill an in memory table with the results of the subquery and hash it on T2.ID, it then just needs to lookup against this hash table for each row in T .

Which behaviour you want really depends on how much data you are expecting from each table/subquery. If you have 1 million rows in T2 , and 10 in T then there is no point in filling a temporary table with 1 million rows, only to subsequently only use it 10 times, whereas if you have a large number of rows in T and only a small amount in T2 the additional cost of materialising the subquery will be beneficial in the long run.

Another thing to point out (which has no impact on performance), the JOIN syntax you are using is the ANSI 89 syntax and was replaced by ANSI 92 explicit JOIN syntax over 20 years ago. Although directed at SQL Server, I think this article summarises the reasons to switch to the newer join syntax very well. Making your final query:

SELECT  *
FROM    product p,
        INNER JOIN store s
            ON p.store = s.idstore
        INNER JOIN branches b
            ON b.idproduct = p.idproduct
        INNER JOIN
        (   SELECT DISTINCT p.common_name
            FROM    shopping_list_content s
                    INNER JOIN product p
                        ON s.iditem = p.idproduct
            WHERE   s.idlist =$listid
        ) s
            ON s.common_name = p.common_name;

NB Most of the above does not apply if you are using MySQL 5.6.5 or later. In this version they introduced more Subquery Optimization that solved a lot of the above issues

Answer 2

This is your query fixed up to use proper join syntax:

select *
from product p join
     store s
     on p.store = s.idstore join
     branches b
     on b.idproduct = p.idproduct
where p.common_name IN (SELECT p.common_name
                        FROM shopping_list_content slc join
                             product p
                             ON slc.iditem = p.idproduct AND
                                slc.idlist = $listid
                       );

Assuming that the same common_name does not appear on multiple products and that shopping_list_content has no duplicate rows, you can replace this with a simple join :

select *
from product p join
     store s
     on p.store = s.idstore join
     branches b
     on b.idproduct = p.idproduct join
     shopping_list_content slc
     on slc.iditem = p.idproduct and
        slc.idlist = $listid;

However, those assumptions may not be true. In that case, changing the subquery to use exists may help performance:

select *
from product p join
     store s
     on p.store = s.idstore join
     branches b
     on b.idproduct = p.idproduct
where exists (SELECT 1
              FROM shopping_list_content slc join
                   product p2
                   on slc.iditem = p2.idproduct AND
                      slc.idlist = $listid
              WHERE p.common_name = p2.common_name
             );

For this latter query, an index on product(common_name, idproduct) along with shopping_list_content(iditem, idlist) should help.