在联合之前消除重复

Question

I need to run a query, that select two columns from a big table (3m+ rows, with selecting two columns, the result set is around 6-7m) and returns a list. So I used union to merge the columns into a list and also to eliminate duplicates. The problem is that I cant return the result in one query, I need to partition it, so I applied a LIMIT ?,? to the subqueries, which the application layer sets via Prepared Statements.

SELECT val
FROM 
(
    (SELECT fs.smr as val
    FROM `fr_search` as fs
    ORDER BY val LIMIT ?,?)

    UNION

    (SELECT fs.dmr as val
    FROM `fr_search` as fs
    ORDER BY val LIMIT ?,?)
) as vals
GROUP BY val

The problem: The union eliminates the duplicates, but only after the LIMIT is applied. Meaning If the two query returns 100+100=200 rows and most of them is a duplicate, I only return <200 rows. How can I apply a limit to such a query, that I can return a specific amount of rows? (If I apply the LIMIT after the subqueries, It will take more than two minutes to run, so It will not solve the problem. )

Answer 1

You don't actually need a subquery for this. The following will work for the first 100 rows:

 (SELECT DISTINCT fs.smr as val
  FROM `fr_search` as fs
  ORDER BY val
  LIMIT 100
 )
 UNION
 (SELECT DISTINCT fs.dmr as val
  FROM `fr_search` as fs
  ORDER BY val
  LIMIT 100
 )
 ORDER BY val
 LIMIT 100;

However, once you start putting in offset, it gets more complicated. For the next 100 rows:

 (SELECT DISTINCT fs.smr as val
  FROM `fr_search` as fs
  ORDER BY val
  LIMIT 200
 )
 UNION
 (SELECT DISTINCT fs.dmr as val
  FROM `fr_search` as fs
  ORDER BY val
  LIMIT 200
 )
 ORDER BY val
 LIMIT 100, 100;

The problem is that you don't know where the second set will come from.

If you actually need to page through the result set, I would suggest that you store it in a temporary table and page off of the temporary table.

Answer 2

Query optimisation is always has two parts to the solution. And is sometimes an iterative process of try, measure and compare.

1. Write a good (and ofc accurate) query that the engine can run efficiently.
2. Ensure the appropriate indexes are available so the optimiser can choose a good execution plan.

The best query is most likely the straight-forward and simple:

SELECT  v.val
FROM    (
        SELECT  fs.smr as val
        FROM    `fr_search` as fs
        UNION
        SELECT  fs.dmr as val
        FROM    `fr_search` as fs
        ) as v
ORDER BY v.val LIMIT ?,?;

In order to run efficiently, you'll want 2 indexes:

• one on fr_search.smr
• the other on fr_search.dmr

---

If the optimiser cannot handle the above, then try using index hints to force it to use the indexes.

In an extreme pinch you could try forcing the issue with the following:

SELECT  v.val
FROM    (
        SELECT  DISTINCT fs.smr as val
        FROM    `fr_search` as fs
        ORDER BY fs.smr LIMIT ?
        UNION
        SELECT  DISTINCT fs.dmr as val
        FROM    `fr_search` as fs
        ORDER BY fs.dmr LIMIT ?
        ) as v
ORDER BY v.val LIMIT ?,?;

Note that your substitutions (assuming pages of 100) should be as follows:

Page 1: 100, 100, 100, 0
Page 2: 200, 200, 100, 100
Page 3: 300, 300, 100, 200
Page 4: 400, 400, 100, 300
etc.

The reason is, you need to cater for a possible imbalance of cross column ordering favouring either table. So for example page 4:

• Get top 400 distinct rows ordered by the key from each column.
• Return rows 301 to 400 of the merged data.
• This could be the last 400 rows of one of the sub-queries. But it's more likely to return about 50 rows from each subquery somewhere above the 150 row mark.

Answer 3

You have two options:

You can SELECT DISTINCT in the inner and outer queries:

SELECT DISTINCT val
FROM 
(
    (SELECT DISTINCT fs.smr as val
    FROM `fr_search` as fs)

    UNION ALL

    (SELECT DISTINCT fs.dmr as val
    FROM `fr_search` as fs)
) as vals
ORDER BY val LIMIT ?,?;

or you can group by your inner queries too, before then grouping by the outer query.

SELECT val
FROM 
(
    (SELECT fs.smr as val
    FROM `fr_search` as fs
    GROUP BY fs.smr)

    UNION ALL

    (SELECT fs.dmr as val
    FROM `fr_search` as fs
    GROUP BY fs.dmr)
) as vals
GROUP BY val
ORDER BY val LIMIT ?,?;

Both will do essentially the same thing in this particular scenario. However in both you should use union all, so that the UNION part doesn't do work on its own, and you are explicit about how you want your record grouping. I would also move the limit clause to the outer query