MySQL Optimization: DISTINCT too slow

Question

I have a query with the form

SELECT COUNT(*)
FROM 
   table1, table2, table3, table4, table5, table6
where (several conditions to join the tables)

Which takes a few minutes to execute (the resulting count is 2000 million).

The problem is that when I add a DISTINCT like here:

SELECT COUNT(  DISTINCT field1, field2, field3, field4, field5   )
FROM 
   table1, table2, table3, table4, table5, table6
where (several conditions to join the tables)

The execution goes to more than 3 hours.

The problem seems to be on the DISTINCT operation, since the JOIN conditions are the same on both versions. Is there any way to optimize the execution of the DISTINCT version?

Thank you!

Answer 1

Which takes a few minutes to execute (the resulting count is 2000 million).

In order to process the count(distinct) , MySQL is going to have to sort 2 BILLION rows. That is a lot of data.

It takes lots and lots of time. There is not much you can do about it, unless you can revise the query so the intermediate data is not so large.

Answer 2

Without changing the SQL statement, or without changing the structure of the tables (for example, adding some indexes, or some new intermediate summary tables), then there's probably no way to get the DISTINCT version of the query to run faster.

Likely the "big rock" in terms of elapsed time is the "Using filesort" operation. We use EXPLAIN to see the query execution plan; that's a quick first step for us to get a handle on what operations MySQL is performing, in what order.

There's some system parameters in MySQL and MariaDB that can be tweaked, for example allocating more memory for temporary sets to be held in MEMORY before spilling to disk; but its unlikely that there's a magic tweak that is going to fix a performance problem with 2 billion rows. And there's always the option of throwing more better hardware at the problem.

If you would be open to changing the SQL statement, or running multiple statements, then there might be an opportunity for tuning.

Answer 3

您是否尝试使用单独具有不同词的新选择语句进行左连接？