How to implement twitter and facebook api like cursor based pagination in mongodb in nodejs using official mongodb client?

Question

I am trying to implement pagination which should not show duplicate data in different pages on data insertion or deletion.

Basic offset-based approach using skip and limit to paginate data is given on this
How to implement pagination for mongodb in node.js using official mongodb client?

Offset-based approach has a big flaw: if the results list has changed between calls to the API, the indices would shift and cause an item to be either returned twice or skipped and never returned

This problem is demonstrated at
https://www.sitepoint.com/paginating-real-time-data-cursor-based-pagination/

Time-based pagination approach would be little better because results are no longer skipped. If you query the first page, and then a new item is deleted, it won't shift the results in your second page and all is fine. However, this approach has a major flaw: what if there is more than one item that was created at the same time?

Offset-based and time-based approaches are not so perfect. So I need the way twitter and facebook api work, which uses cursor based approach, using nodejs and official mongodb client ?

Answer 1

Cursor based pagination can be implemented using any field in collection which is Unique, Orderable and Immutable .

_id satisfy all Unique, Orderable and Immutable conditions. Based on this field we can sort and return page result with _id of last document as the cusror for subsequent request.

curl https://api.mixmax.com/items?limit=2

const items = db.items.find({}).sort({
   _id: -1
}).limit(2);

const next = items[items.length - 1]._id
res.json({ items, next })

when the user wants to get the second page, they pass the cursor (as next) on the URL: curl https://api.mixmax.com/items?limit=2&next=590e9abd4abbf1165862d342

const items = db.items.find({
  _id: { $lt: req.query.next }
}).sort({
   _id: -1
}).limit(2);

const next = items[items.length - 1]._id
res.json({ items, next })

If we want to return results in a different order, such as the date the item then we will add sort=launchDate to the querystring. curl https://api.mixmax.com/items?limit=2&sort=launchDate

const items = db.items.find({}).sort({
   launchDate: -1
}).limit(2);

const next = items[items.length - 1].launchDate;
res.json({ items, next })

For subsequent page request
curl https://api.mixmax.com/items?limit=2&sort=launchDate&next=2017-09-11T00%3A44%3A54.036Z

const items = db.items.find({
  launchDate: { $lt: req.query.next }
}).sort({
   _id: -1
}).limit(2);

const next = items[items.length - 1].launchDate;
res.json({ items, next });

If we launched a bunch of items on the same day and time? Now our launchDate field is no longer unique and doesn't satisfy Unique, Orderable and Immutable . condition. We can't use it as a cursor field. But we could use two fields to generate the cursor.Since we know that the _id field in MongoDB always satisfies the above three condition, we know that if we use it alongside our launchDate field, the combination of the two fields would satisfy the requirements and could be together used as a cursor field. curl https://api.mixmax.com/items?limit=2&sort=launchDate

const items = db.items.find({}).sort({
   launchDate: -1,
  _id: -1 // secondary sort in case there are duplicate launchDate values
}).limit(2);

const lastItem = items[items.length - 1];
// The cursor is a concatenation of the two cursor fields, since both are needed to satisfy the requirements of being a cursor field
const next = `${lastItem.launchDate}_${lastItem._id}`;
res.json({ items, next });

For subsequent page request
curl https://api.mixmax.com/items?limit=2&sort=launchDate&next=2017-09-11T00%3A44%3A54.036Z_590e9abd4abbf1165862d342

const [nextLaunchDate, nextId] = req.query.next.split(‘_’);
const items = db.items.find({
  $or: [{
    launchDate: { $lt: nextLaunchDate }
  }, {
    // If the launchDate is an exact match, we need a tiebreaker, so we use the _id field from the cursor.
    launchDate: nextLaunchDate,
  _id: { $lt: nextId }
  }]
}).sort({
   _id: -1
}).limit(2);

const lastItem = items[items.length - 1];
// The cursor is a concatenation of the two cursor fields, since both are needed to satisfy the requirements of being a cursor field
const next = `${lastItem.launchDate}_${lastItem._id}`;
res.json({ items, next });

Refefence: https://engineering.mixmax.com/blog/api-paging-built-the-right-way/