具有NodeJS驱动程序的MongoDB聚合游标

Question

I am using MongoDB v3.2 and I'm using the native nodejs driver v2.1. When running the aggregation pipeline on large data sets(1mil+ documents), I am encountering the following error:

 'aggregation result exceeds maximum document size (16MB)'

Here is my aggregation pipeline code:

var eventCollection = myMongoConnection.db.collection('events');
var cursor = eventCollection.aggregate([
                {
                    $match: {
                        event_type_id: {$eq: 89012}
                    }
                },
                {
                    $group: {
                        _id: "$user_id",
                        score: {$sum: "$points"}
                    }
                },
                {
                    $sort: {
                        score: -1
                    }
                }
            ],
            {
                cursor: {
                    batchSize: 500
                },
                allowDiskUse: true,
                explain: false
            }, function () {

            });

Things I've tried:

//Using cursor event listeners. None of the on listeners seem to work. Always get error about 16mb.
cursor.on("data", function (data) {
   console.log("Some data: ", data);
});
cursor.on("end", function (data) {
   console.log("End of data: ", data);
});

//Using forEach. Which I thought would allow for >16mb because it's used in conjunction with the batchSize and cursor.
cursor.forEach(function (item) {

})

I've seen in other answers ( How could I write aggregation without exceeds maximum document size? ) that I need to have the results returned by a cursor, so how do I properly do that? I just can't seem to get it to work. Any suggestions on what the batchSize should be?

I am using the native mongodb package - https://github.com/mongodb/node-mongodb-native for a nodejs project not the mongo command line.

Answer 1

Ok I figured it out. It was not working because I was passing in a callback function as the last parameter in the aggregate method. By passing null, it allowed the stream to work as expected. Changes shown below:

var cursor = eventCollection.aggregate([
            {
                $match: {
                    event_type_id: {$eq: 89012}
                }
            },
            {
                $group: {
                    _id: "$user_id",
                    score: {$sum: "$points"}
                }
            },
            {
                $sort: {
                    score: -1
                }
            }
        ],
        {
            cursor: {
                batchSize: 500
            },
            allowDiskUse: true,
            explain: false
        }, null);