Stream query results with the native mongoDB driver for node

Question

I have 100,000 records in a mongoDB collection and trying to retrieve them in a node.js application using the native driver.

I follow the example in MongoDB doc for CursorStream but get the error:

RangeError: Maximum call stack size exceeded

Before this error I get many of these:

(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral.

here is my code:

var query = {...};
var fields = {...};
var options = {
    // "limit": 10
    //"skip": 10,
    //"sort": title
}

var stream = myCollection.find(query, fields, options).stream();
//  stream.pause();
var results = [];
stream.on('data', function (item){
    results.push(item);
    stream.pause();
    // Restart the stream after 1 miliscecond
    setTimeout(function() {
        stream.resume();
    }, 1);
});

stream.on('close'.....

The error occurs also when I don't define a listener for the 'data' event. but it doesn't occur if a pause the stream right after its creation.

The mongod version is v2.4.1 The node driver version is 1.2.x

Any help/hint would be appreciated.

Answer 1

看起来问题是通过在游标流中设置批处理大小来解决的：

var stream = myCollection.find(query, fields, options).batchSize(10000).stream();

Answer 2

You might want to try setImmediate instead of setTimeout, to ensure outstanding IO operations can be flushed. See also https://github.com/ZJONSSON/streamz/issues/1