MongoDB API pagination

Question

Imagine situation when a client has feed of objects with limit 10.
When the next 10 are required it sends request with skip 10 and limit 10.

But what if there are some new objects were added (or deleted) to collection since the 1st request with offset == 0.

Then on 2nd request (with offset == 10) response may have wrong objects order.

Sorting on time of their creation does not work here, because I have some feeds which are formed on sorting via some numeric field.

Answer 1

It really depends on what you want the result to be.

If you want the original objects in their original order regardless of Delete and Add operations then you need to make a copy of the list (or at least of the order) and then page through that. Copy every Id to a new collection that doesn't change once the page has loaded and then paginate through that.

Alternatively, and perhaps more likely, what you want is to see the next 10 after the last one in the current set including any Delete or Add operations that have take place since. For this, you can use the sorted order in which you are viewing them and a filter, $gt whatever the last item was. BUT that doesn't work when there are duplicates in the field on which you are sorting. To get around that you will need to index on that field PLUS some other field which is unique per record, for example, the _id field. Now, you can take the last record in the first set and look for records that are $eq the indexed value and $gt the _id OR are simply $gt the indexed value.

Answer 2

You can add a time field like created_at or updated_at. It must updated when ever the document is created or modified and the field must be unique.

Then query the DB for the range of time using $gte and $lte along with a sort on this time field.

This ensures that any changes made outside the time window will not get reflected in the pagination, provided that the time field does not have duplicates. Most probably if you include microtime, duplicates wont happen.