Select 行在大表中具有唯一值的特定列

Question

table1 has 3 columns in my database: id , timestamp , cluster and it has about 1M rows. I want to query the newest 24 rows with unique cluster value (no row must have repeated cluster value in the returned 24 rows). the usual solution would be:

SELECT
    *
FROM table1
GROUP BY cluster
ORDER BY timestamp DESC
LIMIT 24

however, since I have 1M rows, this query takes so long to be executed. so my solution was to run:

WITH x AS
(
    SELECT
        *
    FROM `table1`
    ORDER BY timestamp DESC
    LIMIT 50
)
SELECT
    *
FROM x
GROUP BY x.cluster
ORDER BY x.timestamp DESC
LIMIT 24

which assumes we can find 24 rows with unique cluster value in every 50 rows. this query runs much faster (~.007 sec). now I want to ask is there any more efficient/routine way for such case?

Answer 1

Your assumption that in the last 50 rows you will find 24 different clusters may not be correct.

Try with ROW_NUMBER() window function:

SELECT *
FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY cluster ORDER BY timestamp DESC) rn
  FROM table1
) t
WHERE rn = 1
ORDER BY timestamp DESC LIMIT 24

Answer 2

You can use row_number() , but you need the right indexes:

select t.*
from (select t.*,
             row_number() over (partition by cluster order by timestamp desc) as seqnum
      from t
     ) t
where seqnum = 1
order by timestamp desc
limit 24;

The index you want is on (cluster, timestamp desc) .

For your purposes, this may still not be sufficient because it is still processing all the rows, even with an index, when you only need a couple of dozen.

I don't know how many recent rows you need to be sure that you have 24 clusters. However, you might find that this works better if we assume that the most recent 1000 rows have at least 24 clusters:

select t.*
from (select t.*,
             row_number() over (partition by cluster order by timestamp desc) as seqnum
      from (select t.*
            from t
            order by timestamp desc
            limit 1000
           ) t
     ) t
where seqnum = 1
order by timestamp desc
limit 24;

For this, you want an index only on (timestamp desc) .

Note: You might find that a where clause on the timestamp works better in this case:

where timestamp > now() - interval 24 hour

for instance to only consider rows in the past 24 hours.

Answer 3

Since you want "one specific cluster value", this will be fast:

SELECT
    *
FROM table1
WHERE cluster = ?
ORDER BY timestamp DESC
LIMIT 24

And have

INDEX(cluster, timestamp)

If that is not what you want, please reword the title and the Question.