How to use mongodb/mongo-go-driver to perform efficient paging

Question

I read in the following article that it is more efficient to use the natural ordering of _id to perform pagination because skip always starts from the beginning of the collection.

Fast and Efficient Pagination in MongoDB

// Page 1
db.students.find().limit(10)

// Page 2
last_id = ...  # logic to get last_id
db.students.find({'_id': {'$gt': last_id}}).limit(10)

But I have no idea how to perform the above using the mongodb/mongo-go-driver .

Answer 1

The cursor.skip() method requires the server to scan from the beginning of the input results set before beginning to return results. As the offset increases, cursor.skip() will become slower. While range queries can use indexes to avoid scanning unwanted documents, typically yielding better performance as the offset grows compared to using cursor.skip() for pagination. See more information on MongoDB: Pagination Example

Using the current version of mongo-go-driver (v0.0.15).An example to perform pagination showing latest entry first:

func Paginate(collection *mongo.Collection, startValue objectid.ObjectID, nPerPage int64) ([]bson.Document, *bson.Value, error) {

    // Query range filter using the default indexed _id field. 
    filter := bson.VC.DocumentFromElements(
        bson.EC.SubDocumentFromElements(
            "_id",
            bson.EC.ObjectID("$gt", startValue),
        ),
    )

    var opts []findopt.Find
    opts = append(opts, findopt.Sort(bson.NewDocument(bson.EC.Int32("_id", -1))))
    opts = append(opts, findopt.Limit(nPerPage))

    cursor, _ := collection.Find(context.Background(), filter, opts...)

    var lastValue *bson.Value
    var results []bson.Document
    for cursor.Next(context.Background()) {
        elem := bson.NewDocument()
        err := cursor.Decode(elem)
        if err != nil {
            return results, lastValue, err
        }
        results = append(results, *elem)
        lastValue = elem.Lookup("_id")
    }

    return results, lastValue, nil
}

An example to call the pagination function above:

database := client.Database("databaseName")
collection := database.Collection("collectionName")
startObjectID, _ := objectid.FromHex("5bbafea2b5e14ee3a298fa4a")

// Paginate only the latest 20 documents 
elements, lastID, err := Paginate(collection, startObjectID, 20)
for _, e := range elements {
    fmt.Println(&e)
}
// Last seen ObjectID can be used to call next Paginate() 
fmt.Println("Last seen ObjectID: ", lastID.ObjectID())

Note that you can also substitute the _id field with another indexed field.

Answer 2

you can create a new func, dont forget to pass http.writer to read parameter.

func Pagination(r *http.Request, FindOptions *options.FindOptions) (int64, int64) {
    if r.URL.Query().Get("page") != "" && r.URL.Query().Get("limit") != "" {
        page, _ := strconv.ParseInt(r.URL.Query().Get("page"), 10, 32)
        limit, _ := strconv.ParseInt(r.URL.Query().Get("limit"), 10, 32)
        if page == 1 {
            FindOptions.SetSkip(0)
            FindOptions.SetLimit(limit)
            return page, limit
        }

        FindOptions.SetSkip((page - 1) * limit)
        FindOptions.SetLimit(limit)
        return page, limit

    }
    FindOptions.SetSkip(0)
    FindOptions.SetLimit(0)
    return 0, 0
}

just call

Pagination(r, options)

example

options := options.Find()
page, limit := parameter.Pagination(r, options)
// page, limit as response for header payload