I am using MongoDB v3.2 and I'm using the native nodejs driver v2.1. When running the aggregation pipeline on large data sets(1mil+ documents), I am encountering the following error:
'aggregation result exceeds maximum document size (16MB)'
Here is my aggregation pipeline code:
var eventCollection = myMongoConnection.db.collection('events');
var cursor = eventCollection.aggregate([
{
$match: {
event_type_id: {$eq: 89012}
}
},
{
$group: {
_id: "$user_id",
score: {$sum: "$points"}
}
},
{
$sort: {
score: -1
}
}
],
{
cursor: {
batchSize: 500
},
allowDiskUse: true,
explain: false
}, function () {
});
Things I've tried:
//Using cursor event listeners. None of the on listeners seem to work. Always get error about 16mb.
cursor.on("data", function (data) {
console.log("Some data: ", data);
});
cursor.on("end", function (data) {
console.log("End of data: ", data);
});
//Using forEach. Which I thought would allow for >16mb because it's used in conjunction with the batchSize and cursor.
cursor.forEach(function (item) {
})
I've seen in other answers ( How could I write aggregation without exceeds maximum document size? ) that I need to have the results returned by a cursor, so how do I properly do that? I just can't seem to get it to work. Any suggestions on what the batchSize should be?
I am using the native mongodb package - https://github.com/mongodb/node-mongodb-native for a nodejs project not the mongo command line.
Ok I figured it out. It was not working because I was passing in a callback function as the last parameter in the aggregate method. By passing null, it allowed the stream to work as expected. Changes shown below:
var cursor = eventCollection.aggregate([
{
$match: {
event_type_id: {$eq: 89012}
}
},
{
$group: {
_id: "$user_id",
score: {$sum: "$points"}
}
},
{
$sort: {
score: -1
}
}
],
{
cursor: {
batchSize: 500
},
allowDiskUse: true,
explain: false
}, null);
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.