WHERE子句中的MySQL RAND（）匹配一小组行

Question

i get into an interesting issue with MySQL. When i try to multiply RAND() function with some big integer, i get max random number really small. Here is my MySQL query, that should be a very fast random query, but it returns ID max 36000, even when there are 4600000+ IDs.

SET @maxID=(SELECT MAX(id) FROM property); #it's about 4600000

SELECT * FROM property
WHERE 
downloaded_at IS NULL
AND id >= FLOOR(1 + RAND() * @maxID) #this returns max +/-36000
LIMIT 100

When i move this code into plain SELECT query, everything is fine

SELECT FLOOR(1 + RAND() * (SELECT MAX(id) FROM property))

Could someone please explain, why this error occurs? Thank you!

edits

---

Hm, somehow when i remove downloaded_at IS NULL it comes to sences, ID's are higher, but results are not that random anymore.

---

I can't use ORDER BY RAND(), because table is too big, query is too slow and whole server crashes eventually in few minutes

---

version is 5.7.21-0ubuntu0.16.04.1

Answer 1

Your random row selection method is biased ... the probabilty of a row being seleted is proportional to its id. Eg if you had 10 rows with id = 1 to 10 then 1 has 10% chance of being selected, 2 has 20% and so on.

Also, the reason why your code selects ids less than ~36000 is obvious: rows are (usually) processed in PK order and by the time 100th matching row is found, the query has only processed row with id of around 36000.

Now, if you are interested in selecting 100 random rows, you can use this query instead:

SELECT *
FROM property
WHERE id IN (
    SELECT id
    FROM property
    WHERE downloaded_at IS NULL
    ORDER BY RAND()
    LIMIT 100
)

Or may be this (rough outline):

SELECT *
FROM property
WHERE id IN (
    SELECT id
    FROM property
    WHERE RAND() <= 100.0 / @maxID -- explanation below
    LIMIT 100
)

The above does not involve sorting but it still needs to scan all ids. 100.0 is same as the desired number of rows but add some more just to be sure. This should result in equal probability for each row to be selected.

Answer 2

The problem is that rand() is being called every time the condition in the where clause is evaluated. Instead, put the value in a subquery:

SELECT p.*
FROM property p CROSS JOIN
     (SELECT FLOOR(1 + RAND() * @maxID) as idlim) x
WHERE p.downloaded_at IS NULL AND
      p.id >= x.idlim #this returns max +/-36000
LIMIT 100;

This ensures that the rand() function is called only once.