在MySql中通过限制来优化订单

Question

I have a 3 million records table called "transactions" .

CREATE TABLE transactions(
  id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  lookupAId int(6) NOT NULL,
  .....
  updateDate TIMESTAMP
)

In the worst case the user will specify no filters and the query would looks like this :

select * from transactions
   join lookupA on (well indexed columns) 
   .. ( 12 lookup table joins) 
order by updateDate limit 500

Without the order by clause the query runs in milliseconds, but with the order by it takes about a minute. The table is projected to grow to 12-15 million records.

1. My SLA is to get results in under a second, is it possible in MySql ?
2. How can I optimize the order by clause to make this perform.

I run MySql 5.7 in xLarge memory optimized RDS instance in AWS

UPDATE 1 updateDate has a time component and is indexed (B-tree, non-unique)

Update 2 This worked , although I don't know why

SELECT * FROM (select * from transactions order by updateDate) transactions
   join lookupA on (well indexed columns) 
   .. ( 12 lookup table joins) 
   limit 500

Answer 1

如果您还没有的话，那么ORDER BY肯定会从索引中受益：

create index ix1 on transactions (updateDate);

Answer 2

MySQL is probably doing a lot of work on the query before limiting the query size with limit. This seems to be a known weakness of MySQL.

Try doing the select from transactions in a subquery to limit the result set size before doing the joins.

SELECT * FROM (select * from transactions order by updateDate limit 500) transactions
   join lookupA on (well indexed columns) 
   .. ( 12 lookup table joins) 

Answer 3

The usual technique for tackling this problem:

SELECT ... JOIN ...
    LIMIT ...

is to:

1. Do the minimal amount of work to find the PRIMARY KEY values of the rows that factor into the LIMIT rows.
2. Feed those ids into the JOINs to get the rest of the info.

As your query stands, the Optimizer throws up its hands and simply does all the JOIN (optimizing each as best it can), generating a large (many rows, many columns) intermediate table, then apply the ORDER BY (sort the many rows of many columns) and LIMIT (deliver some of those rows).

With INDEX(OrderDate) (and that column is in the table it chooses to start the JOINing with) the Optimizer can at least consider using the index. But that might be the worst case -- What if there are not 500 rows to be had; it will have done all the work anyway!

Answer 4

The Optimizer does not know that a table is a simple "lookup" table. It must be prepared to find 0 rows or more than 1 row.

Case 1: You know that there is exactly 1 row in each of the lookup ( JOINed ) tables:

Case 2: You know that there is at most 1 row in each lookup table.

In both of these cases, the following is an efficient way to rewrite the query:

SELECT  t.a, t.b, ...
        ( SELECT name FROM LU1 WHERE id = t.name_id ) AS name, 
        ( SELECT foo  FROM LU1 WHERE id = t.foo_id ) AS foo, 
        ...
    FROM transactions AS t
    ORDER BY t.OrderDate
    LIMIT ...

and

INDEX(OrderDate)
INDEX(id)  -- for each LU table, unless there is already `PRIMARY KEY(id)`

This formulation of the query will focus on walking through exactly 500 rows, presorted by OrderDate , looking up 12 things for each row.

It is semantically equivalent to Case 2 ( LEFT JOIN ) since it gives NULL for name (etc) when there is no mapping.

Technically, Case 1 is not the same. If a lookup fails, JOIN will fail to count the row, but my reformulation will keep the row, showing NULL .