MongoDB REST API dynamic data pagination

Question

I am implementing REST API with Express and Mongoose. I am stuck with a task to paginate results for a collection with large number of documents. Document is a social media post which has fields like: created_at , updated_at and score .

Ordinary paging implementation using limit and page number doesn't work because the nature of data. Posts can be removed or added while user is browsing the results.

I could implement range-based pagination as suggested in MongoDB documentation. Last items in a page created_at value could be used to query next page.

Post.find({ created_at: { $gt: req.query.next } })

But what if I want to sort results by score which is not unique or all scores are the same?

Answer 1

There are multiple ways to do this, but in your case i would suggest to create the find / sort conditions dynamicly and use them in your find statement.

Outta my head just to give you an idea:

var MyFilter = {};
var MySort = {};
if (req.query.created_at != undefined) {
    MyFilter = { created_at : { $gte : req.query.created_at) } };
}

if (req.query.sortOn != undefined){
    MySort = { req.query.sort : 1 };
}

Post.find(MyFilter)
.sort(MySort)
.exec(function (err, posts) {
    ===do your stuff===
});

Answer 2

https://groups.google.com/forum/#!searchin/mongodb-user/paging%7Csort:relevance/mongodb-user/_Jrv9vI1lDE/Wg0eGV2AkZYJ

I just found the same topic in google groups with perfect answer.

You could try creating a compound index on age and _id, and sort by both fields to get the first three documents. To get the subsequent set of users, your query should return documents that satisfy either of the following requirements:

• have the same age as the last document and have a greater _id or
• have a greater age

For example (lastAge and lastID are the age and _id of the latest user):

db.collection.find( { $or : [ { age : lastAge, _id : {$gt : lastID } } , { age : {$gt : lastID } } ] } )