Pagination Offset Issues - MySQL

Question

I have an orders grid holding 1 million records. The page has pagination, sort and search options. So If the sort order is set by customer name with a search key and the page number is 1, it is working fine.

SELECT * FROM orders WHERE customer_name like '%Henry%' ORDER BY 
customer_name desc limit 10 offset 0

It becomes a problem when the User clicks on the last page.

SELECT * FROM orders WHERE customer_name like '%Henry%' ORDER BY 
customer_name desc limit 10 offset 100000

The above query takes forever to load. Index is set to the order id, customer name, date of order column.

I can use this solution https://explainextended.com/2009/10/23/mysql-order-by-limit-performance-late-row-lookups/ if I don't have a non-primary key sort option, but in my case sorting is user selected. It will change from Order id, customer name, date of order etc.

Any help would be appreciated. Thanks.

Answer 1

Problem 1:

LIKE "%..." -- The leading wildcard requires a full scan of the data, or at least until it finds the 100000+10 rows. Even

... WHERE ... LIKE '%qzx%' ... LIMIT 10

is problematic, since there probably not 10 such names. So, a full scan of your million names.

... WHERE name LIKE 'James%' ...

will at least start in the middle of the table-- if there is an index starting with name . But still, the LIMIT and OFFSET might conspire to require reading the rest of the table.

Problem 2: (before you edited your Question!)

If you leave out the WHERE , do you really expect the user to page through a million names looking for something?

This is a UI problem.

If you have a million rows, and the output is ordered by Customer_name, that makes it easy to see the Aarons and the Zywickis, but not anyone else. How would you get to me (James)? Either you have 100K links and I am somewhere near the middle, or the poor user would have to press [Next] 'forever'.

My point is that the database is not the place to introduce efficiency.

In some other situations, it is meaningful to go to the [Next] (or [Prev]) page. In these situations, "remember where you left off", then use that to efficiently reach into the table. OFFSET is not efficient. More on Pagination

Answer 2

I use a special concept for this. First I have a table called pager . It contains an primary pager_id, and some values to identify a user (user_id,session_id), so that the pager data can't be stolen.

Then I have a second table called pager_filter . I consist of 3 ids:

pager_id     int unsigned not NULL  # id of table pager
order_id     int unsigned not NULL  # store the order here
reference_id int unsigned not NULL  # reference into the data table
primary key(pager_id,order_id);

As first operation I select all records matching the filter rules from and insert them into pager_filter

DELETE FROM pager_filter WHERE pager_id = $PAGER_ID;

INSERT INTO pager_filter (pager_id,order_id,reference_id)
SELECT $PAGER_ID pager_id, ROW_NUMBER() order_id, data_id reference_id
FROM   data_table
WHERE  $CONDITIONS
ORDER BY $ORDERING

After filling the filter table you can use an inner join for pagination:

SELECT  d.*
FROM    pager_filter f
INNER JOIN data_table d ON d.data_id = f.reference id
WHERE   f.pager_id = $PAGER_ID && f.order_id between 100000 and 100099
ORDER BY f.order_id

or

SELECT  d.*
FROM    pager_filter f
INNER JOIN data_table d ON d.data_id = f.reference id
WHERE   f.pager_id = $PAGER_ID
ORDER BY f.order_id
LIMIT 100 OFFSET 100000

Hint: All code above is not tested pseudo code