OR in MySQL where makes query very slow

Question

I have a big query in MySQL for magento which takes way to much time. I tried to optimize it but my MySQL knowledge isn't enough to get it solved, so maybe someone could take a look and give me some hints in the right direction.

Select Distinct
  `e`.*,
  `cat_index`.`position` AS `cat_index_position`,
  `price_index`.`price`,
  `price_index`.`tax_class_id`,
  `price_index`.`final_price`,
  IF(price_index.tier_price IS NOT NULL, 
      LEAST(price_index.min_price, price_index.tier_price),
      price_index.min_price) AS `minimal_price`,
  `price_index`.`min_price`,
  `price_index`.`max_price`,
  `price_index`.`tier_price`
From
  `catalog_product_entity` AS `e`
    Inner Join
  `catalog_category_product_index` As `cat_index`
    On cat_index.product_id=e.entity_id And
       cat_index.store_id=1 And
       cat_index.visibility In(2, 4) And
       cat_index.category_id='2'
    Inner Join
  `catalog_product_index_price` AS `price_index` 
    On price_index.entity_id = e.entity_id And
       price_index.website_id = '1' And
       price_index.customer_group_id = 0
    Left Join
  `beta_entity_product` AS `beta` 
    On e.entity_id = beta.product_id And
       beta.entity_id In (81558, 81559, ... stupidly long list of ids)
    Left Join 
  `catalog_product_entity_int` AS `is_uni` 
    On e.entity_id = is_uni.entity_id And
       attribute_id = 179 
Where 
 is_uni.value = 1 OR 
 beta.product_id IS NOT NULL

If I just have 1 condition in the WHERE clause everything is fine, but with the OR it takes sometimes some minutes to finish and thats way too long. Which options do I have to get better results? Another problem is that I can't make more queries from this and just join the results together. Everything has to be in 1 query.

When I do an EXPLAIN on the query I get the following result (copied in JSON format for better overview):

{
        "data":
        [
            {
                "id": 1,
                "select_type": "SIMPLE",
                "table": "e",
                "type": "ALL",
                "possible_keys": "PRIMARY",
                "key": null,
                "key_len": null,
                "ref": null,
                "rows": 213396,
                "Extra": "Using temporary"
            },
            {
                "id": 1,
                "select_type": "SIMPLE",
                "table": "beta",
                "type": "range",
                "possible_keys": "PRIMARY",
                "key": "PRIMARY",
                "key_len": "4",
                "ref": null,
                "rows": 2833,
                "Extra": "Using where; Using index"
            },
            {
                "id": 1,
                "select_type": "SIMPLE",
                "table": "is_uni",
                "type": "ref",
                "possible_keys": "UNQ_CATALOG_PRODUCT_ENTITY_INT_ENTITY_ID_ATTRIBUTE_ID_STORE_ID,IDX_CATALOG_PRODUCT_ENTITY_INT_ATTRIBUTE_ID,IDX_CATALOG_PRODUCT_ENTITY_INT_ENTITY_ID",
                "key": "UNQ_CATALOG_PRODUCT_ENTITY_INT_ENTITY_ID_ATTRIBUTE_ID_STORE_ID",
                "key_len": "6",
                "ref": "unc_cpk.e.entity_id,const",
                "rows": 1,
                "Extra": "Using where"
            },
            {
                "id": 1,
                "select_type": "SIMPLE",
                "table": "cat_index",
                "type": "eq_ref",
                "possible_keys": "PRIMARY,IDX_CAT_CTGR_PRD_IDX_PRD_ID_STORE_ID_CTGR_ID_VISIBILITY,15D3C269665C74C2219037D534F4B0DC",
                "key": "PRIMARY",
                "key_len": "10",
                "ref": "const,unc_cpk.e.entity_id,const",
                "rows": 1,
                "Extra": "Using where"
            },
            {
                "id": 1,
                "select_type": "SIMPLE",
                "table": "price_index",
                "type": "eq_ref",
                "possible_keys": "PRIMARY,IDX_CATALOG_PRODUCT_INDEX_PRICE_CUSTOMER_GROUP_ID,IDX_CATALOG_PRODUCT_INDEX_PRICE_WEBSITE_ID",
                "key": "PRIMARY",
                "key_len": "8",
                "ref": "unc_cpk.cat_index.product_id,const,const",
                "rows": 1,
                "Extra": "Using where"
            }
        ]
    }

Answer 1

If each query with one condition is individually fast, try doing

Select
  blah
Where
  is_uni.value = 1
Union
Select
  blah
Where
  beta.product_id Is Not Null

As long as they don't return too many results, this will also be reasonably quick.

Or when you said quick with one condition, did you just mean the first?

Answer 2

It looks like you need an index on beta.product_id unless this is your primary key there?

Otherwise, you can do this as a Union query each one having a different where clause:

SELECT DISTINCT `e`.*, `cat_index`.`position` AS `cat_index_position`, `price_index`.`price`, 
   `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT 
   NULL, LEAST(price_index.min_price, price_index.tier_price),
    price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, 
   `price_index`.`max_price`, `price_index`.`tier_price` 
FROM `catalog_product_entity` AS `e`
 INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id 
      AND cat_index.store_id=1 AND cat_index.visibility IN(2, 4) AND cat_index.category_id='2'
 INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id 
  AND price_index.website_id = '1' AND price_index.customer_group_id = 0
 LEFT JOIN `beta_entity_product` AS `beta` ON e.entity_id = beta.product_id AND beta.entity_id IN (81558,81559,84592,84593,87758,87759,87760,87761,90944,90945,90946,90947,94364,94365,94366,94367,98458,98459,98460,98461,98462,98463,104194,104195,104196,104197,110151,110152,110153,110154,110155,110156,116583,116584,116585,116586,123366,123367,123368,123369,123370,123371,123372,123373,130500,130501,130502,130503,130504,130505,130506,138075,138076,138077,138078,138079,138080,146179,146180,146181,146182,146183,146184,154543,154544,154545,154546,154547,154548,154549,154550,163639,163640,163641,163642,163643,163644,163645,173998,173999,174000,174001,174002,174003,174004,174005,174006,174007,184545,184546,184547,184548,184549,184550,195249,195250,195251,195252,195253,195254,195255,195256,195257,195258,206400,206401,206402,206403,206404,206405,217632,217633,217634,217635,217636,217637,217638,217639,229029,229030,229031,229032,229033,229034,229035,229036,229037,229038,229039,229040,240350,240351,240352,240353,240354,240355,240356,240357,240358,240359,240360,240361,251631,251632,251633,251634,251635,251636,251637,251638,251639,262900,262901,262902,262903,262904,262905,262906,262907,262908,262909,274205,274206,274207,274208,274209,274210,274211,274212,274213,274214,274215,274216,274217,285747,285748,285749,285750,285751,285752,285753,285754,285755,285756,285757,285758,297747,297748,297749,297750,297751,297752,297753,297754,297755,297756,297757,297758,297759,297760,297761,309660,309661,309662,309663,309664,309665,309666,309667,322314,322315,322316,322317,322318,322319,322320,322321,334814,334815,334816,334817,334818,334819,334820,334821,334822,334823,334824,334825,334826,346810,346811,346812,346813,346814,346815,346816,346817,346818,346819,346820,346821,358550,358551,358552,358553,358554,358555,358556,358557,358558,358559,358560,358561,358562,370376,370377,370378,370379,370380,370381,370382,370383,370384,370385,370386,370387,381938,381939,381940,381941,381942,381943,381944,381945,381946,381947,381948,381949,381950,391081,391082,391083,391084,391085,391086,391087,391088,391089,391090,391091,391092,391093,396111,396112,396113,396114,396115,396116,396117,396118,396119,396120,396121,396122,396123,396124,396125)
 LEFT JOIN `catalog_product_entity_int` AS `is_uni` ON e.entity_id = is_uni.entity_id AND attribute_id = 179 


WHERE is_uni.value = 1 
UNION
SELECT DISTINCT `e`.*, `cat_index`.`position` AS `cat_index_position`, `price_index`.`price`, 
   `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT 
   NULL, LEAST(price_index.min_price, price_index.tier_price),
    price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, 
   `price_index`.`max_price`, `price_index`.`tier_price` 
FROM `catalog_product_entity` AS `e`
 INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id 
      AND cat_index.store_id=1 AND cat_index.visibility IN(2, 4) AND cat_index.category_id='2'
 INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id 
  AND price_index.website_id = '1' AND price_index.customer_group_id = 0
 LEFT JOIN `beta_entity_product` AS `beta` ON e.entity_id = beta.product_id AND beta.entity_id IN (81558,81559,84592,84593,87758,87759,87760,87761,90944,90945,90946,90947,94364,94365,94366,94367,98458,98459,98460,98461,98462,98463,104194,104195,104196,104197,110151,110152,110153,110154,110155,110156,116583,116584,116585,116586,123366,123367,123368,123369,123370,123371,123372,123373,130500,130501,130502,130503,130504,130505,130506,138075,138076,138077,138078,138079,138080,146179,146180,146181,146182,146183,146184,154543,154544,154545,154546,154547,154548,154549,154550,163639,163640,163641,163642,163643,163644,163645,173998,173999,174000,174001,174002,174003,174004,174005,174006,174007,184545,184546,184547,184548,184549,184550,195249,195250,195251,195252,195253,195254,195255,195256,195257,195258,206400,206401,206402,206403,206404,206405,217632,217633,217634,217635,217636,217637,217638,217639,229029,229030,229031,229032,229033,229034,229035,229036,229037,229038,229039,229040,240350,240351,240352,240353,240354,240355,240356,240357,240358,240359,240360,240361,251631,251632,251633,251634,251635,251636,251637,251638,251639,262900,262901,262902,262903,262904,262905,262906,262907,262908,262909,274205,274206,274207,274208,274209,274210,274211,274212,274213,274214,274215,274216,274217,285747,285748,285749,285750,285751,285752,285753,285754,285755,285756,285757,285758,297747,297748,297749,297750,297751,297752,297753,297754,297755,297756,297757,297758,297759,297760,297761,309660,309661,309662,309663,309664,309665,309666,309667,322314,322315,322316,322317,322318,322319,322320,322321,334814,334815,334816,334817,334818,334819,334820,334821,334822,334823,334824,334825,334826,346810,346811,346812,346813,346814,346815,346816,346817,346818,346819,346820,346821,358550,358551,358552,358553,358554,358555,358556,358557,358558,358559,358560,358561,358562,370376,370377,370378,370379,370380,370381,370382,370383,370384,370385,370386,370387,381938,381939,381940,381941,381942,381943,381944,381945,381946,381947,381948,381949,381950,391081,391082,391083,391084,391085,391086,391087,391088,391089,391090,391091,391092,391093,396111,396112,396113,396114,396115,396116,396117,396118,396119,396120,396121,396122,396123,396124,396125)
 LEFT JOIN `catalog_product_entity_int` AS `is_uni` ON e.entity_id = is_uni.entity_id AND attribute_id = 179
WHERE beta.product_id IS NOT NULL

Depending on your data and if the OR is mutually exclusive data, you could use an UNION ALL. The Query you are running is not able to fully take advantage of an INDEX MERGE that would be required to make it run fast with the OR statement.

Answer 3

There are multiple solutions:

1) Split query and use UNION - possible sorting and pagination problems 2) Add new property that will be used only for this filter so instead of

is_uni.value = 1 OR beta.product_id IS NOT NULL

you would be querying only

feature_filter = 1

3) Decompose query into multiple queries - I would suggest this as first step since queries as big as this one are really great way to shoot yourself in the foot

BTW. Using distinct on query with so many columns eats a lot of resources (mysql needs to compare each column value!)

Answer 4

Suggestions:

1.) Instead of

beta.product_id IS NOT NULL

write

not (beta.product_id IS NULL)

2.) You can create a stored procedure or a stored function if you want a single request for your query. Of course you can use multiple queries and at the end merge the partial results using a single database server request.

3.) If you have complex db logics, use an ORM , like Flourish .

I wish you good luck.