Simple SQL query not using index with order clause and larger limit MariaDB

Question

I just came across an issue with database not taking in account index when selecting larger amount of data.

It's a simple select with order clause.

This query (or any other with limit less then a million)

SELECT * 
FROM mon_stat_detail 
WHERE 1 
ORDER BY id DESC
LIMIT 500000

properly uses an index on column id (btw. its not primary but unique index)

While this query

SELECT *
FROM mon_stat_detail 
WHERE 1 
ORDER BY id DESC
LIMIT 1000000

is using file sort.

Table is quite large, it has about 60 mil. records.

With file sort it takes 15 minutes and creates over 20GB of data on disk due to filesort

However if I force index on the same query

SELECT * 
FROM mon_stat_detail FORCE INDEX (id_2) 
WHERE 1 
ORDER BY id DESC
LIMIT 1000000

it is using it and takes just seconds as expected.

Any idea why is this happening? Why do I need to force this index on such a simple query?

(pared down schema:)

CREATE TABLE mon_stat_detail (
    id int(16) unsigned NOT NULL AUTO_INCREMENT, 
    sensor_id int(10) unsigned NOT NULL, 
    time datetime NOT NULL, 
    … other about 10 columns ... 
    PRIMARY KEY (sensor_id,time), 
    UNIQUE KEY id_2 (id), 
    … some more indexes and FK …
) ENGINE=InnoDB AUTO_INCREMENT=550579790 DEFAULT CHARSET=utf8

Database:

• Server version: 10.1.48-MariaDB-0+deb9u2 - Debian 9.13
• Protocol version: 10

Answer 1

• If the goal is to fetch any million rows, leave out the ORDER BY .

• If the goal is to fetch the last million rows and there are no gaps in id , then use

   WHERE id > (SELECT MAX(id) FROM mon_stat_detail) - 1000000 ORDER BY id ASC -- note; and no LIMIT is needed

• If there might be gaps, you could see if running this (before the Select) helps:

   ANALYZE TABLE mon_stat_detail;

For InnoDB it is fast and it refreshes the "statistics" on which the query plan is based.