分别保留记录的子集以提高MySQL的查询性能

Question

I have a large table containing over 10 million records and It will keep growing. I am performing an aggregation query (count of particular value) on records of last 24 hours. The time taken by this query will keep increasing with number of records in the table.

I can limit the time taken by keeping these 24 hours records in separate table and perform aggregation on that table. Does mysql provide any functionality to handle this kind of scenario?

Table schema and query for reference:

CREATE TABLE purchases (
    Id int(11) NOT NULL AUTO_INCREMENT, 
    ProductId int(11) NOT NULL, 
    CustomerId int(11) NOT NULL, 
    PurchaseDateTime datetime(3) NOT NULL, 
    PRIMARY KEY (Id), 
    KEY ix_purchases_PurchaseDateTime (PurchaseDateTime) USING BTREE, 
    KEY ix_purchases_ProductId (ProductId) USING BTREE, 
    KEY ix_purchases_CustomerId (CustomerId) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

select COALESCE(sum(ProductId = v_ProductId), 0),
       COALESCE(sum(CustomerId = v_CustomerId), 0)
    into v_ProductCount, v_CustomerCount
    from purchases
    where PurchaseDateTime > NOW() - INTERVAL 1 DAY
      and (   ProductId = v_ProductId
           or CustomerId = v_CustomerId );

Answer 1

Build and maintain a separate Summary table .

With partitioning, you might get a small improvement, or you might get no improvement. With a summary table, you might get a factor of 10 improvement.

The summary table could have a 1-day resolution, or you might need 1-hour. Please provide SHOW CREATE TABLE for what you currently have, so we can discuss more specifics.

(There is no built-in mechanism for what you want.)

Answer 2

Plan A

I would leave off

      and (   ProductId = v_ProductId
           or CustomerId = v_CustomerId )

since the rest of the query will simply deal with it anyway.

Then I would add

INDEX(PurchaseDateTime, ProductId, CustomerId)

which would be "covering" -- that is, the entire SELECT can be performed in the INDEX's BTree. It would also be 'clustered' in the sense that all the data needed would be consecutively stored in the index. Yes, the datetime is deliberately first. ( OR is a nuisance to optimize. I don't trust the Optimizer to do "index merge union".)

Plan B

If you expect to touch very few rows (because of v_ProductId and v_CustomerId ), then the following may be faster, in spite of being more complex:

SELECT COALESCE(sum(ProductId = v_ProductId), 0)
    INTO v_ProductCount
    FROM purchases
    WHERE PurchaseDateTime > NOW() - INTERVAL 1 DAY
      AND ProductId = v_ProductId;
SELECT COALESCE(sum(CustomerId = v_CustomerId), 0)
    INTO v_CustomerCount
    FROM purchases
    WHERE PurchaseDateTime > NOW() - INTERVAL 1 DAY
      AND CustomerId = v_CustomerId;

together with both:

INDEX(ProductId, PurchaseDateTime),
INDEX(CustomerId, PurchaseDateTime)

Yes, the order of the columns is deliberately different.

Original Question

Both of these approaches are better than your original suggestion of a separate table. These isolate the data in one part of an index (or two indexes), thereby having the effect of "separate". And these do the task with less effort on your part.