如何让这个 MySQL 查询更快更好？

Question

I have this query which I am trying to make faster as of right now - it times-out before unless I change WHERE customers.total_orders >= 1 to WHERE customers.total_orders = 1.

Essentially the query is supposed to show the total number of customers that placed by customer for 1st, 2nd, 3rd, 4th, 5th, and 6th time and the average value of the order per. The time in-between is the average time it takes before the customer places another order.

The question I am trying to answer is if a customer ordered X product, what is their lifetime value.

SELECT

  count(o1_id) as "Total-O1",
  avg(o1_total) as "Order Value-O1",
  avg(TimeDiff_o1) as "Avg-o1-o2(Days)",

  count(o2_id) as "Total-o2",
  avg(o2_total) as "Order Value-o2",
  avg(TimeDiff_o2) as "Avg-o2-o3(Days)",

  count(o3_id) as "Total-o3",
  avg(o3_total) as "Order Value-o3",
  avg(TimeDiff_o3) as "Avg-o3-o4(Days)",

  count(o4_id) as "Total-o4",
  avg(o4_total) as "Order Value-o4",
  avg(TimeDiff_o4) as "Avg-o4-o5(Days)",

  count(o5_id) as "Total-o5",
  avg(o5_total) as "Order Value-o5"

FROM (
   SELECT 
       o1.id as o1_id,
       o1.mc_gross AS o1_total,
       timestampdiff(DAY,o1.purchased_at,o2.purchased_at) AS TimeDiff_o1,

       o2.id as o2_id,
       o2.mc_gross AS o2_total,
       timestampdiff(DAY,o2.purchased_at,o3.purchased_at) AS TimeDiff_o2,

       o3.id as o3_id,
       o3.mc_gross AS o3_total,
       timestampdiff(DAY,o3.purchased_at,o4.purchased_at) AS TimeDiff_o3,

       o4.id as o4_id,
       o4.mc_gross AS o4_total,
       timestampdiff(DAY,o4.purchased_at,o5.purchased_at) AS TimeDiff_o4,

       o5.id as o5_id,
       o5.mc_gross AS o5_total,
       timestampdiff(DAY,o5.purchased_at,o6.purchased_at) AS TimeDiff_o5,

       o6.id as o6_id,
       o6.mc_gross AS o6_total

    FROM customers

    cross join orders as o1 on o1.customer_id = customers.id and o1.store_id = 10 and customers.created_at >= curdate() - interval 365 day

    cross join order_items o1_order_items ON o1_order_items.order_id = o1.id and o1_order_items.product_variant_id = 1

    LEFT JOIN orders o2 ON o2.customer_id = customers.id
        AND o2.store_id = 10
        AND o2.parent_order_id = 0
        AND o2.id != o1.id
    LEFT JOIN orders o3 ON o3.customer_id = customers.id
        AND o3.store_id = 10
        AND o3.parent_order_id = 0
        AND o3.id != o1.id
        AND o3.id != o2.id
    LEFT JOIN orders o4 ON o4.customer_id = customers.id
        AND o4.store_id = 10
        AND o4.parent_order_id = 0
        AND o4.id != o1.id
        AND o4.id != o2.id
        AND o4.id != o3.id
    LEFT JOIN orders o5 ON o5.customer_id = customers.id
        AND o5.store_id = 10
        AND o5.parent_order_id = 0
        AND o5.id != o1.id
        AND o5.id != o2.id
        AND o5.id != o3.id
        AND o5.id != o4.id
    LEFT JOIN orders o6 ON o6.customer_id = customers.id
        AND o6.store_id = 10
        AND o6.parent_order_id = 0
        AND o6.id != o1.id
        AND o6.id != o2.id
        AND o6.id != o3.id
        AND o6.id != o4.id
        AND o6.id != o5.id

    WHERE customers.total_orders >= 1

    group by customers.id

) as a

CREATE TABLE `customers` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `address_city` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_country` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_country_code` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_name` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_firstname` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_state` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_status` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_street` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_zip` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `first_name` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
  `last_name` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
  `instagram_username` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `payer_business_name` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `payer_email` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `payer_id` varchar(13) COLLATE utf8_unicode_ci DEFAULT NULL,
  `payer_status` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `contact_phone` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `sanitized_phone` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `sanitized_phone_last_10` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
  `residence_country` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
  `username` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
  `password` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `first_order_id` int(11) DEFAULT '0',
  `last_order_id` int(11) DEFAULT '0',
  `total_orders` int(11) DEFAULT '0',
  `total_gross` decimal(8,2) DEFAULT '0.00',
  `avatar` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'http://lorempixel.com/200/200/',
  `oauth_avatar` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `auth_type` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
  `subscr_id` varchar(19) COLLATE utf8_unicode_ci DEFAULT NULL,
  `type` enum('default','wholesale') COLLATE utf8_unicode_ci NOT NULL DEFAULT 'default',
  `verified` tinyint(1) DEFAULT NULL,
  `verify_token` varchar(256) COLLATE utf8_unicode_ci DEFAULT NULL,
  `remember_token` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `last_login_at` timestamp NULL DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `deleted_at` timestamp NULL DEFAULT NULL,
  `promotion_opt_in` tinyint(1) NOT NULL DEFAULT '0',
  `marketing_sms_opt_in` tinyint(1) DEFAULT NULL,
  `referral_code` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
  `date_of_birth` date DEFAULT NULL,
  `updated_on_st_at` timestamp NULL DEFAULT NULL,
  `send_survey_email` tinyint(1) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `customers_address_city_index` (`address_city`),
  KEY `customers_address_country_index` (`address_country`),
  KEY `customers_address_country_code_index` (`address_country_code`),
  KEY `customers_address_state_index` (`address_state`),
  KEY `customers_address_zip_index` (`address_zip`),
  KEY `customers_first_name_index` (`first_name`),
  KEY `customers_last_name_index` (`last_name`),
  KEY `customers_payer_email_index` (`payer_email`),
  KEY `customers_residence_country_index` (`residence_country`),
  KEY `customers_contact_phone_index` (`contact_phone`),
  KEY `customers_payer_business_name_index` (`payer_business_name`),
  KEY `customers_address_street_index` (`address_street`),
  KEY `customers_address_name_index` (`address_name`),
  KEY `deleted_at` (`deleted_at`),
  KEY `promotion_opt_in` (`promotion_opt_in`),
  KEY `type` (`type`),
  KEY `first_order_id` (`first_order_id`),
  KEY `sanitized_phone` (`sanitized_phone`),
  KEY `sanitized_phone_last_10` (`sanitized_phone_last_10`)
) ENGINE=InnoDB AUTO_INCREMENT=1361589 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

CREATE TABLE `orders` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `warehouse_id` int(11) DEFAULT NULL,
  `external_order_id` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
  `whs_order_id` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
  `txn_id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `customer_id` int(11) NOT NULL,
  `address_id` int(11) DEFAULT NULL,
  `payment_profile_id` int(11) DEFAULT NULL,
  `payment_status_id` int(11) NOT NULL,
  `shipping_service_id` int(11) DEFAULT NULL,
  `shipping_service_option_id` int(11) DEFAULT NULL,
  `pricing_cost_id` int(11) DEFAULT NULL,
  `subscription_group_id` int(11) DEFAULT NULL,
  `fulfillment_note` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
  `fulfillment_status_id` int(10) unsigned NOT NULL,
  `fulfillment_status_note` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
  `fulfilled_at` timestamp NULL DEFAULT NULL,
  `shipment_status_id` int(11) NOT NULL,
  `order_status_id` int(11) NOT NULL,
  `agent_id` int(11) NOT NULL,
  `store_id` int(11) NOT NULL,
  `parent_order_id` int(11) NOT NULL,
  `custom` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `memo` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `tax` decimal(9,2) DEFAULT NULL,
  `order_status_url` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `status_short_url` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `payment_date` varchar(28) COLLATE utf8_unicode_ci DEFAULT NULL,
  `shipping_method` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
  `shipping` varchar(9) COLLATE utf8_unicode_ci DEFAULT NULL,
  `shipping_priority` tinyint(4) DEFAULT NULL,
  `exchange_rate` varchar(9) COLLATE utf8_unicode_ci DEFAULT NULL,
  `mc_currency` varchar(3) COLLATE utf8_unicode_ci DEFAULT NULL,
  `mc_fee` double(8,2) DEFAULT '0.00',
  `mc_gross` double(8,2) DEFAULT '0.00',
  `mc_handling` double(8,2) DEFAULT '0.00',
  `mc_shipping` double(8,2) DEFAULT '0.00',
  `mc_discount` double(8,2) DEFAULT '0.00',
  `mc_store_credit` double(8,2) DEFAULT '0.00',
  `total` double(8,2) DEFAULT NULL,
  `address_name` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_email` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_company` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_street` varchar(200) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_city` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_state` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_zip` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_country_code` varchar(2) COLLATE utf8_unicode_ci DEFAULT NULL,
  `address_phone` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `fulfillment_provider_id` int(11) DEFAULT NULL,
  `fulfillment_provider_order_id` int(11) DEFAULT NULL,
  `ship_after` timestamp NULL DEFAULT NULL,
  `confirmed_at` timestamp NULL DEFAULT NULL,
  `agentupdated_at` timestamp NULL DEFAULT NULL,
  `purchased_at` timestamp NULL DEFAULT NULL,
  `created_at` timestamp NULL DEFAULT NULL,
  `updated_at` timestamp NULL DEFAULT NULL,
  `deleted_at` timestamp NULL DEFAULT NULL,
  `delay_notice_sent_at` timestamp NULL DEFAULT NULL,
  `cost_of_goods_sold` decimal(8,2) NOT NULL DEFAULT '0.00',
  `use_store_credit` tinyint(1) DEFAULT NULL,
  `recurring_shipping_option` tinyint(1) DEFAULT NULL,
  `send_shipping_updates` tinyint(1) DEFAULT NULL,
  `bypass_address_validation` tinyint(1) DEFAULT NULL,
  `address_updated_at` timestamp NULL DEFAULT NULL,
  `held_by_user_id` int(11) DEFAULT NULL,
  `sent_to_klaviyo_at` timestamp NULL DEFAULT NULL,
  `updated_by_user_id` int(11) DEFAULT NULL,
  `created_by_user_id` int(11) DEFAULT NULL,
  `subscription` tinyint(1) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `orders_txn_id_index` (`txn_id`),
  KEY `orders_customer_id_index` (`customer_id`),
  KEY `orders_payment_status_id_index` (`payment_status_id`),
  KEY `orders_shipment_status_id_index` (`shipment_status_id`),
  KEY `orders_order_status_id_index` (`order_status_id`),
  KEY `orders_agent_id_index` (`agent_id`),
  KEY `orders_store_id_index` (`store_id`),
  KEY `orders_payment_date_index` (`payment_date`),
  KEY `orders_shipping_method_index` (`shipping_method`),
  KEY `orders_shipping_index` (`shipping`),
  KEY `orders_exchange_rate_index` (`exchange_rate`),
  KEY `orders_mc_currency_index` (`mc_currency`),
  KEY `orders_mc_fee_index` (`mc_fee`),
  KEY `orders_mc_gross_index` (`mc_gross`),
  KEY `orders_mc_handling_index` (`mc_handling`),
  KEY `orders_mc_shipping_index` (`mc_shipping`),
  KEY `orders_external_order_id_index` (`external_order_id`) USING BTREE,
  KEY `orders_whs_order_id_index` (`whs_order_id`) USING BTREE,
  KEY `orders_fulfillment_status_id_index` (`fulfillment_status_id`) USING BTREE,
  KEY `orders_parent_order_id_index` (`parent_order_id`) USING BTREE,
  KEY `orders_custom_index` (`custom`) USING BTREE,
  KEY `orders_address_name_index` (`address_name`) USING BTREE,
  KEY `orders_address_email_index` (`address_email`) USING BTREE,
  KEY `orders_address_company_index` (`address_company`) USING BTREE,
  KEY `orders_address_city_index` (`address_city`) USING BTREE,
  KEY `orders_address_state_index` (`address_state`) USING BTREE,
  KEY `orders_address_zip_index` (`address_zip`) USING BTREE,
  KEY `orders_address_country_code_index` (`address_country_code`) USING BTREE,
  KEY `orders_fulfillment_provider_id_index` (`fulfillment_provider_id`) USING BTREE,
  KEY `orders_fulfillment_provider_order_id_index` (`fulfillment_provider_order_id`) USING BTREE,
  KEY `orders_confirmed_at_index` (`confirmed_at`) USING BTREE,
  KEY `orders_agentupdated_at_index` (`agentupdated_at`) USING BTREE,
  KEY `address_id` (`address_id`),
  KEY `payment_profile_id` (`payment_profile_id`),
  KEY `shipping_service_id` (`shipping_service_id`),
  KEY `subscription_group_id` (`subscription_group_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1950715 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

CREATE TABLE `order_items` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `order_id` int(11) NOT NULL,
  `parent_id` int(10) unsigned DEFAULT NULL,
  `product_variant_id` int(11) NOT NULL,
  `subscription_plan_id` int(11) DEFAULT NULL,
  `subscription_id` int(11) DEFAULT NULL,
  `created_by_user_id` int(11) DEFAULT NULL,
  `updated_by_user_id` int(11) DEFAULT NULL,
  `item_name` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `item_number` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `quantity` varchar(127) COLLATE utf8_unicode_ci DEFAULT NULL,
  `type` enum('default','post_purchase','subscription','free','fbt') COLLATE utf8_unicode_ci NOT NULL DEFAULT 'default',
  `unit_price` decimal(9,2) DEFAULT NULL,
  `mc_gross` decimal(9,2) DEFAULT NULL,
  `mc_handling` decimal(9,2) DEFAULT NULL,
  `mc_shipping` decimal(9,2) DEFAULT NULL,
  `tax` decimal(9,2) DEFAULT NULL,
  `free_item` tinyint(4) DEFAULT NULL,
  `replaced_item_id` int(11) DEFAULT NULL,
  `replacement_item` tinyint(4) DEFAULT NULL,
  `athlete_item` tinyint(4) DEFAULT NULL,
  `agent_request` tinyint(4) DEFAULT NULL,
  `care_package` tinyint(4) DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `deleted_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `order_items_order_id_index` (`order_id`),
  KEY `order_items_product_variant_id_index` (`product_variant_id`),
  KEY `order_items_item_name_index` (`item_name`),
  KEY `order_items_item_number_index` (`item_number`),
  KEY `order_items_quantity_index` (`quantity`),
  KEY `order_items_mc_gross_index` (`mc_gross`),
  KEY `order_items_mc_handling_index` (`mc_handling`),
  KEY `order_items_mc_shipping_index` (`mc_shipping`),
  KEY `order_items_tax_index` (`tax`),
  KEY `order_items_agent_request_index` (`agent_request`),
  KEY `order_items_care_package_index` (`care_package`),
  KEY `order_items_athlete_item_index` (`athlete_item`),
  KEY `order_items_free_item_index` (`free_item`),
  KEY `order_items_replacement_item_index` (`replacement_item`),
  KEY `parent_id` (`parent_id`),
  KEY `deleted_at` (`deleted_at`),
  KEY `subscription_id` (`subscription_id`)
) ENGINE=InnoDB AUTO_INCREMENT=6675238 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Here is the explain table.

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	PRIMARY		ALL					841782
2	DERIVED	o1_order_items	ref	order_items_order_id_index,order_items_product_variant_id_index	order_items_product_variant_id_index	4	const	841782	Using temporary; Using filesort
2	DERIVED	o1	eq_ref	PRIMARY,orders_customer_id_index,orders_store_id_index	PRIMARY	4	shop.o1_order_items.order_id	1	Using where
2	DERIVED	customers	eq_ref	PRIMARY,customers_address_city_index,customers_address_country_index,customers_address_country_code_index,customers_address_state_index,customers_address_zip_index,customers_first_name_index,customers_last_name_index,customers_payer_email_index,customers_residence_country_index,customers_contact_phone_index,customers_payer_business_name_index,customers_address_street_index,customers_address_name_index,deleted_at,promotion_opt_in,type,first_order_id,sanitized_phone,sanitized_phone_last_10	PRIMARY	4	shop.o1.customer_id	1	Using where
2	DERIVED	o2	ref	orders_customer_id_index,orders_store_id_index,orders_parent_order_id_index	orders_customer_id_index	4	shop.customers.id	1	Using where
2	DERIVED	o3	ref	orders_customer_id_index,orders_store_id_index,orders_parent_order_id_index	orders_customer_id_index	4	shop.customers.id	1	Using where
2	DERIVED	o4	ref	orders_customer_id_index,orders_store_id_index,orders_parent_order_id_index	orders_customer_id_index	4	shop.customers.id	1	Using where
2	DERIVED	o5	ref	orders_customer_id_index,orders_store_id_index,orders_parent_order_id_index	orders_customer_id_index	4	shop.customers.id	1	Using where
2	DERIVED	o6	ref	orders_customer_id_index,orders_store_id_index,orders_parent_order_id_index	orders_customer_id_index	4	shop.customers.id	1	Using where

Answer 1

• Start with finding the Customer ids meeting `customer for 1st, 2nd, 3rd, 4th, 5th, and 6th time and the average value of the order per". The goal here is to do the minimal work needed to find those ids.

• Get the count of orders (1-6) and min and max date (see below). Later we will "pivot" them into multiple columns (see below).

• After that, JOIN to other (or the same) tables to fetch the other desired columns. The hope here is that the previous step whittled down the number of ids so much that this step does not have to do as much work.

• Eventually "pivot" to turn rows into columns. This is how to deal with the multiple sets of

  count(o3_id) as "Total-o3", avg(o3_total) as "Order Value-o3", avg(TimeDiff_o3) as "Avg-o3-o4(Days)",

• Compute the avg time in between the way nnichols did -- (max-min)/count. No need for LAG(), which may be quite costly.

• This may involve Selects nested to a depth of 2 or 3, and will have a few JOINs , but probably no LEFT JOINs .

Answer 2

Your current query structure cannot work! Your query returns when you have customers.total_orders = 1 because none of the left joins returns a set. As an example take one customer with 10 orders but only one containing the desired product. 1st three tables return 1 row, 1st LEFT JOIN returns 9 rows, 2nd 8 rows, 3rd 7 rows, 4th 6 rows and 5th 5 rows. That's a set of 9x8x7x6x5=15,120 (before applying GROUP BY) for 1 customer. If customer placed 20 orders it would be 1,395,360 rows. Adding an index to total_orders will not fix this.

In your question you asked -

if a customer ordered X product, what is their lifetime value

so the solution is to find all customers who have ordered product X and then join back to orders to find their lifetime value -

SELECT
    c1.*,
    COUNT(o.id) AS num_orders,
    SUM(o.mc_gross) AS orders_total,
    MIN(o.purchased_at) AS first_order,
    MAX(o.purchased_at) AS last_order,
    IF(COUNT(*) = 1, 0, ROUND(TIMESTAMPDIFF(DAY, MIN(o.purchased_at), MAX(o.purchased_at)) / (COUNT(o.id) - 1))) AS avg_between_orders
FROM (
    SELECT customers.id, customers.name
    FROM customers
    INNER JOIN orders o
        ON o.customer_id = customers.id AND o.store_id = 10
    INNER JOIN order_items oi
        ON oi.order_id = o.id AND oi.product_variant_id = 1
    # THE INNER JOIN to orders renders this WHERE clause redundant
    # WHERE customers.total_orders >= 1
    WHERE customers.created_at >= CURDATE() - INTERVAL 365 DAY
    GROUP BY customers.id
) AS c1
INNER JOIN orders o ON c1.id = o.customer_id
GROUP BY c1.id

My tiny test dataset (3 customers, 13 orders and 3 order_items)shows 36,134 rows examined for your query but 16 rows examined with my query.