How to optimize performance inside MySQL Query filtered via NOT IN or NOT EXISTS

Question

One query what I have perform low performances by adding one query filter.

Here is full query:

SELECT
    `c`.`categories_id`,
    `c`.`section_id`,
    `c`.`categories_status`,
    IF(`c`.`categories_status` = 1, 'ON', 'OFF') AS `categories_status_name`,
    TRIM(`cd`.`categories_name`) AS `categories_name`,
    IF(`cd`.`concert_date` <> '',
        DATE_FORMAT(STR_TO_DATE(`cd`.`concert_date`,'%d/%m/%Y'),'%d.%m.%Y'),
        NULL
    ) AS `concert_date`,
    TRIM(`cd`.`concert_time`) AS `concert_time`
FROM
    `categories` `c`
    JOIN `categories_description` `cd` ON `c`.`categories_id` = `cd`.`categories_id` 
WHERE
    `c`.`plan_id` > 2
 AND 
    `c`.`categories_status` = '1' 
 AND 
    `cd`.`categories_id` NOT IN(
    SELECT
        `p`.`parent_id`
    FROM
        `products` `p`
    WHERE
        `p`.`product_type` = 'X'
    AND
        `p`.`parent_id` = `cd`.`categories_id`
    GROUP BY `p`.`product_type`
) 
GROUP BY `c`.`categories_id`
ORDER BY DATE_FORMAT(STR_TO_DATE(`cd`.`concert_date`,'%d/%m/%Y'),'%Y-%m-%d') DESC, `cd`.`categories_name` DESC

Inside this query I have one new filter what is added what looks like this:

 `cd`.`categories_id` NOT IN(
        SELECT
            `p`.`parent_id`
        FROM
            `products` `p`
        WHERE
            `p`.`product_type` = 'X'
        AND
            `p`.`parent_id` = `cd`.`categories_id`
        GROUP BY `p`.`product_type`
    )

I also try one more solution using NOT EXISTS but that is mutch worse:

NOT EXISTS (
     SELECT
        DISTINCT 1
    FROM
        `products` `p`
    WHERE
        `p`.`product_type` = 'X'
    AND
        `p`.`parent_id` = `cd`.`categories_id`
    GROUP BY `p`.`product_type`
)   

My main problem is that after I add this filter for removing categories what contain X products, performances start to be realy bad. Without this filter page loading is arround 0.5-0.8 seconds but with this filter page load can be from 8 to 10 seconds.

Can anyone help me to optimize this query?

Answer 1

This could work. Most engines aren't very good in NOT IN/NOT EXISTS, unless they internally modify the query to the form below. At least it's worth a try.

SELECT
    `c`.`categories_id`,
    `c`.`section_id`,
    `c`.`categories_status`,
    IF(`c`.`categories_status` = 1, 'ON', 'OFF') AS `categories_status_name`,
    TRIM(`cd`.`categories_name`) AS `categories_name`,
    IF(`cd`.`concert_date` <> '',
        DATE_FORMAT(STR_TO_DATE(`cd`.`concert_date`,'%d/%m/%Y'),'%d.%m.%Y'),
        NULL
    ) AS `concert_date`,
    TRIM(`cd`.`concert_time`) AS `concert_time`
FROM
    `categories` `c` JOIN `categories_description` `cd` 
                     ON `c`.`categories_id` = `cd`.`categories_id`
     LEFT JOIN  `products` `p`
     ON `p`.`parent_id` = `cd`.`categories_id`
     AND `p`.`product_type` = 'X'
WHERE
    `c`.`plan_id` > 2
 AND 
    `c`.`categories_status` = '1' 
 AND 
    `p`.`parent_id` IS NULL 
GROUP BY `c`.`categories_id`
ORDER BY DATE_FORMAT(STR_TO_DATE(`cd`.`concert_date`,'%d/%m/%Y'),'%Y-%m-%d') DESC, `cd`.`categories_name` DESC

Answer 2

Here is the most fastest solution I got.

SELECT
    `c`.`categories_id`,
    `c`.`section_id`,
    `c`.`categories_status`,
    IF(`c`.`categories_status` = 1, 'ON', 'OFF') AS `categories_status_name`,
    TRIM(`cd`.`categories_name`) AS `categories_name`,
    IF(`cd`.`concert_date` <> '',
        DATE_FORMAT(STR_TO_DATE(`cd`.`concert_date`,'%d/%m/%Y'),'%d.%m.%Y'),
        NULL
    ) AS `concert_date`,
    TRIM(`cd`.`concert_time`) AS `concert_time`
FROM
    `categories` `c`
    INNER JOIN `categories_description` `cd` ON `c`.`categories_id` = `cd`.`categories_id`
    LEFT JOIN `products` `p` ON `p`.`parent_id` = `cd`.`categories_id`
WHERE
    `c`.`plan_id` > 2
 AND 
    `c`.`categories_status` = '1'
 AND `p`.`product_type` != 'X'
GROUP BY `c`.`categories_id`
ORDER BY DATE_FORMAT(STR_TO_DATE(`cd`.`concert_date`,'%d/%m/%Y'),'%Y-%m-%d') DESC, `cd`.`categories_name` DESC

Thanks to @NigelRen who give me idea how to solve this problem. @Ronald give almost same solution but is a bit slower (0.400 seconds) than my solution.

Thank you guys for help!