MySQL subquery optimisation

Question

I have 4 tables in minimal view here:

Sales:

id
has_discount
discount_is_percentage
discount_amount
**sale_date_time**
**order_status**

Sales_items:

id
**sales_id**
has_discount
discount_is_percentage
discount_amount
**product_id** (This can sometimes be null)
price_inc_vat_per_item
quantity
vat_rate
is_removed

Sales_payments:

id
**sales_id**
payment_amount
payment_change
payment_method

Products:

id
product_name

I have a query which calculates the discount on the fly and report on it. This works great where the total number of records stayed below 100-200k. But as the number increasing, the time taken is really slow. I guess that's because of the subquery I am using. Anyone could shed a light on this please. There is a client_id and outlet_id on each table that distinguish them from other users in the system.

Currently the tables have 1-3 million rows and the client in question have 300k-600k. The query takes 30+ seconds. For others with low amount of rows can get it in even sub-seconds. The ones with stars are the indices. How can the query be improved to get the same desired results? The query I have now:

SELECT  DATE_FORMAT(CONVERT_TZ(sales.sale_date_time,'UTC','Europe/London'),
                '%l%p') as title, count(*) as total_sales, SUM(sales_items.quantity
                   ) as total_quantities,
        SUM(sales_items.price_before_line_discount) as price_before_line_discount,
        SUM(sales_items.price_before_line_discount-sales_items.line_discount) as price_after_line_discount,
        SUM(sales_items.vat_rated_sales) as vat_rated_sales_before_discount,
        SUM(sales_items.zero_rated_sales) as zero_rated_sales_before_discount,
        SUM(sales_items.total_vat_only) as total_vat_only_before_discount,
        SUM(sales_payments.payment_taken) as payment_taken, SUM(sales_items.line_discount) as total_line_discount,
        SUM(sales_payments.payment_cash) as payment_cash, SUM( CASE WHEN sales.has_discount=1
              AND  sales.discount_is_percentage=0 THEN sales.discount_amount WHEN sales.has_discount=1
              AND  sales.discount_is_percentage=1 THEN ((sales_items.price_before_line_discount-sales_items.line_discount)*sales.discount_amount/100) WHEN sales.has_discount=0 THEN 0 END 
           )as total_sales_discount,
        SUM( CASE WHEN sales.has_discount=1 THEN CASE WHEN discount_is_percentage=0 THEN (sales_items.vat_rated_sales*sales.discount_amount)/(sales_items.price_before_line_discount-sales_items.line_discount) WHEN discount_is_percentage=1 THEN (sales_items.vat_rated_sales*((sales_items.price_before_line_discount-sales_items.line_discount)*sales.discount_amount/100))/(sales_items.price_before_line_discount-sales_items.line_discount) END ELSE 0 END )as vat_rated_sales_discount,
        SUM( CASE WHEN sales.has_discount=1 THEN CASE WHEN discount_is_percentage=0 THEN (sales_items.zero_rated_sales*sales.discount_amount)/(sales_items.price_before_line_discount-sales_items.line_discount) WHEN discount_is_percentage=1 THEN ((sales_items.zero_rated_sales*((sales_items.price_before_line_discount-sales_items.line_discount)*sales.discount_amount/100))/(sales_items.price_before_line_discount-sales_items.line_discount)) END ELSE 0 END )as zero_rated_sales_discount,
        SUM( CASE WHEN sales.has_discount=1 THEN CASE WHEN discount_is_percentage=0 THEN (sales_items.total_vat_only*sales.discount_amount)/(sales_items.price_before_line_discount-sales_items.line_discount) WHEN discount_is_percentage=1 THEN (sales_items.total_vat_only*((sales_items.price_before_line_discount-sales_items.line_discount)*sales.discount_amount/100))/(sales_items.price_before_line_discount-sales_items.line_discount) END ELSE 0 END )as total_vat_only_discount
    FROM  `sales`
    left join  
    (
        SELECT  sales_id, SUM(quantity) as quantity, SUM(price_inc_vat_per_item*quantity) AS price_before_line_discount,
                SUM( CASE WHEN has_discount=1
                      AND  discount_is_percentage=0 THEN discount_amount WHEN has_discount=1
                      AND  discount_is_percentage=1 THEN ((price_inc_vat_per_item*quantity)*discount_amount/100) WHEN has_discount=0 THEN 0 END 
                   )as line_discount,
                SUM( CASE WHEN vat_rate>0 THEN CASE WHEN has_discount=1
                      AND  discount_is_percentage=0 THEN ((price_inc_vat_per_item*quantity)-discount_amount) WHEN has_discount=1
                      AND  discount_is_percentage=1 THEN ((price_inc_vat_per_item*quantity)-((price_inc_vat_per_item*quantity)*discount_amount/100)) WHEN has_discount=0 THEN (price_inc_vat_per_item*quantity) END ELSE 0 END 
                   )as vat_rated_sales,
                SUM( CASE WHEN vat_rate=0 THEN CASE WHEN has_discount=1
                      AND  discount_is_percentage=0 THEN ((price_inc_vat_per_item*quantity)-discount_amount) WHEN has_discount=1
                      AND  discount_is_percentage=1 THEN ((price_inc_vat_per_item*quantity)-((price_inc_vat_per_item*quantity)*discount_amount/100)) WHEN has_discount=0 THEN (price_inc_vat_per_item*quantity) END ELSE 0 END 
                   )as zero_rated_sales,
                SUM( CASE WHEN vat_rate>0 THEN CASE WHEN has_discount=1
                      AND  discount_is_percentage=0 THEN ((price_inc_vat_per_item*quantity)-discount_amount)-((price_inc_vat_per_item*quantity)-discount_amount)/(1+(vat_rate/100)) WHEN has_discount=1
                      AND  discount_is_percentage=1 THEN ((price_inc_vat_per_item*quantity)-((price_inc_vat_per_item*quantity)*discount_amount/100))-((price_inc_vat_per_item*quantity)-((price_inc_vat_per_item*quantity)*discount_amount/100))/(1+(vat_rate/100)) WHEN has_discount=0 THEN (price_inc_vat_per_item*quantity)-(price_inc_vat_per_item*quantity)/(1+(vat_rate/100)) END ELSE 0 END 
                   )as total_vat_only
            FROM  sales_items
            WHERE  client_id='0fe26d93-775f-440c-a119-13cbcb6cbc0c'
              AND  is_removed=0
            GROUP BY  sales_id 
    ) as sales_items  ON `sales`.`id` = `sales_items`.`sales_id`
    left join  
    (
        SELECT  sales_id, SUM(payment_amount-payment_change) payment_taken,
                SUM(CASE WHEN payment_method='CASH' THEN (payment_amount-payment_change) ELSE 0 END) as payment_cash
            FROM  sales_payments
            WHERE  client_id='0fe26d93-775f-440c-a119-1396c36cbc0c'
            GROUP BY  sales_id
    ) as sales_payments  ON `sales`.`id` = `sales_payments`.`sales_id`
    WHERE  `sales`.`client_id` = '0fe26d93-775f-440c-a119-1396c36cbc0c'
      and  `sales`.`outlet_id` = 'd5b74bdf-5cef-4455-bf99-13cbcb6cbc0c'
      and  `sales`.`order_status` = 'COMPLETED'
      and  `sale_date_time` >= '2016-01-28 00:00:00'
      and  `sale_date_time` <= '2016-11-28 23:59:00'
    GROUP BY  HOUR(CONVERT_TZ(sales.sale_date_time,'UTC','Europe/London'))
    ORDER BY  `sale_date_time` ASC

UPDATE:

To answer the questions by @rick-james

• I Need to sort it by sale_date_time which is a datetime field. Group by is needed to report by by hour. It also has days, Month-year etc dependig on the period queried.
• Had to use UUID because of the design. The whole DB is around 8GB where these four tables have most of it. The index length is bigger than the actual data size as I had lots of foreign key contraint.

It's on Amazon Aurora with 15GB RAM.

Sales Table: 0.5GB Data 1.3GB Index

Sales Items: 1.3GB Data 3.2GB Index

Sales Payments: 0.5GB Data 1.1GB Index

All tables collation is utf8_unicode_ci.

• It's using Aurora 5.6 which is MySQL 5.6. Here is the explain select.

ID select_type tables type possible_keys keys key_len ref rows filtered extra

1 PRIMARY sales ref sales_client_id_outlet_id_foreign,sales_client_id_index,sales_outlet_id_index,sales_sale_date_time_index,sales_order_status_index sales_client_id_index 108 const 5352 Using index condition; Using where; Using temporary; Using filesort

1 PRIMARY ref 108 MyDB.sales.id 10

1 PRIMARY ref 108 MyDB.sales.id 10

3 DERIVED sales_payments ref sales_payments_client_id_outlet_id_foreign,sales_payments_client_id_index sales_payments_client_id_outlet_id_foreign 108 const 5092 Using index condition; Using where; Using temporary; Using filesort

2 DERIVED sales_items ref sales_items_client_id_outlet_id_foreign,sales_items_client_id_index sales_items_client_id_outlet_id_foreign 108 const 13340 Using index condition; Using where; Using temporary; Using filesort

2 DERIVED products eq_ref PRIMARY,products_id_unique PRIMARY 108 MyDB.sales_items.product_id 1

• May be Will look into store the result in DB and get from there. Only problem is that the old orders can be amended and the total will need to be rebuilt if that happens.

Any other way to rewrite the query to get the desired result?

Answer 1

• When ORDER BY unnecessarily differs from GROUP BY , an extra sort pass is needed.
• UUIDs are terribly inefficient when the data is bigger than can be cached in RAM. How big are the tables? What is the value of `innodb_buffer_pool_size? How much RAM do you have?
• LEFT JOIN ( SELECT ... ) is terribly inefficient until at least 5.6. Please provide EXPLAIN SELECT ... to see if it is Optimized. What version are you using?
• Even worse is LEFT JOIN ( SELECT ... ) LEFT JOIN ( SELECT ... ) . Added: Since I don't see "auto-key" this is bad. It makes me wonder if it is really MySQL 5.6.
• Building and maintaining a "Summary table" may be the ultimate answer. It would probably have a PRIMARY KEY including client_id, outlet_id, order_status, and sale_HOUR.
• Does either subquery run slowly by itself? If so, start a separate question to focus on just the subquery. Please provide output from SHOW CREATE TABLE ; there are a lot of details missing from your description of the tables -- indexes, datatypes, sizes, collations, etc. Added: Still need this; there are still some things to check. A possible solution: CREATE TEMPORARY TABLE with each of the two LEFT JOIN SELECTs ; then use them.