MySQL, WHERE NOT IN very slow query?

Question

SELECT product_id
FROM ebay_shop
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id
WHERE ebay_shop.validated = 2007
AND amazon_shop.validated = 2007

Result(= 150 products): products are validated in ebay_shop && amazon_shop in year 2007

SELECT ebay_shop.product_id
FROM ebay_shop
WHERE ebay_shop.validated = 2010

Result(= 4000 products): products are validated in ebay_shop in year 2010

Question: i want to find products which aren't validated to 2010 of 2007 list, products have in 2007 but not in 2010.

My query:

SELECT product_id
FROM ebay_shop
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id
WHERE ebay_shop.validated = 2007
AND amazon_shop.validated = 2007
AND NOT IN (SELECT ebay_shop.product_id
            FROM ebay_shop
            WHERE ebay_shop.validated = 2010)

My query is very slow(~60s) 150 find in 4000 records. I need help to improve this query.

Answer 1

NOT IN is terrible on performance. Probably the best way is to do a LEFT JOIN for your 2010 and then qualify it is NULL... Any record that IS found, throw it out, just keep the ones that return NULL via the join qualifier.

In addition, ensure you have an index on your ebay shop and amazon shops based on YEAR AND PRODUCT via index ( validated, product_id ). Reason to have the year first, to keep all the common year as small set, then cycle all products within that. Then, the index would also be optimized for the respective join too.

SELECT STRAIGHT_JOIN
      es.product_id
   FROM 
      ebay_shop eshop
         LEFT JOIN amazon_shop ashop
             ON eshop.validated = ashop.validated
            AND eshop.product_id = ashop.product_id

         LEFT JOIN ebay_shop ebay2
            on ebay2.validated = 2010
           and ebay2.product_id = eshop.product_id
   WHERE 
          eshop.validated = 2007
      and ebay2.validated is null

Answer 2

I realize this isn't really an answer but more of a different study case!

I also use "NOT IN" but in a different context. I get the result I what but the query takes more than 10 seconds for a table with about 300 records.

Here is the query:

(
SELECT (
'inscrit'
) AS 
TYPE , membre_mandataire.id AS id, membres.prenom AS prenom, membres.nom_de_famille AS nom_de_famille, membre_mandataire.photo AS photo, membre_mandataire.niveau_qualification AS niveau_qualification, membre_mandataire.numero_mandataire AS numero_mandataire
FROM membres, membre_mandataire
WHERE membre_mandataire.id_membre = membres.id
AND membre_mandataire.confirme = 'Y'
AND membre_mandataire.visible_dans_liste = 'Y'
)
UNION (

SELECT (
'actif'
) AS 
TYPE , mandataires_actifs.id AS id, mandataires_actifs.prenom AS prenom, mandataires_actifs.nom_de_famille AS nom_de_famille, mandataires_actifs.photo AS photo, mandataires_actifs.niveau_qualification AS niveau_qualification, mandataires_actifs.numero_mandataire AS numero_mandataire
FROM mandataires_actifs, membre_mandataire
WHERE (
mandataires_actifs.numero_mandataire
) NOT 
IN (

SELECT membre_mandataire.numero_mandataire
FROM membre_mandataire
WHERE membre_mandataire.confirme = 'Y'
AND membre_mandataire.visible_dans_liste = 'Y'
)
)
ORDER BY nom_de_famille ASC

The field "numero_mandataire" is the constant in both tables. In the result, i Want all records from the table membre_mandataire and UNION or JOIN all records where "numero_mandataire" cannot be found in the table mandataires_actifs .

I guess the solution for me would be to go with "LEFT JOIN" instead of "UNION" but I am still stuck with "NOT IN". I've been trying for a while and I can't make it work.

I am learning MySQL but looks like i'm not quite there yet!

Thanks for any help!

• Blaise

Answer 3

@DRapp wrote a fine workaround and I want to add an other idea...

SELECT group_concat(product_id)
FROM ebay_shop 
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id 
WHERE ebay_shop.validated = 2007 
AND amazon_shop.validated = 2007 

and then:

SELECT product_id    
FROM ebay_shop    
LEFT JOIN amazon_shop ON ebay_shop.product_id = amazon_shop.product_id    
WHERE ebay_shop.validated = 2007    
AND amazon_shop.validated = 2007    
AND NOT IN (1,2,3,4)    /* results from first query */