MongoClient节点游标流和数据管道

Question

I am just starting to learn about node streams, and I am using MongoClient ( MongoClient Cursor Doc ). Within this document it states that I can get a returned query as a stream of documents. Like so:

var MongoClient = require('mongodb').MongoClient
  , assert = require('assert');

// Connection URL
var url = 'mongodb://localhost:27017/myproject';
// Use connect method to connect to the Server
MongoClient.connect(url, function(err, db) {
  assert.equal(null, err);
  console.log("Connected correctly to server");

  var col = db.collection('streams');
  // Insert a single document
  col.insert([{a:1}, {a:1}, {a:1}], function(err, r) {
    assert.equal(null, err);
    assert.equal(3, r.result.n);

    // Get the results using a find stream
    var cursor = col.find({});
    cursor.on('data', function(doc) {
      console.dir(doc);
    });

    cursor.once('end', function() {
      db.close();
    });
  });
});

Now I am trying to use the stream created by var cursor = col.find({}); to pipe into through2 and took out the listeners on data and end like so:

  var cursor = col.find({});

  cursor.pipe(through2(function (buf, _, next) {
    console.log('chunkString: ', buf.toString());
    next();
  }));

However, I get this error:

/Users/blah/projects/mybuz/blah-ad/node_modules/mongodb/lib/utils.js:97
    process.nextTick(function() { throw err; });
                                        ^
TypeError: Invalid non-string/buffer chunk
    at validChunk (/Users/blah/projects/mybuz/blah-ad/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:164:14)

Dont know what I am doing incorrectly because I am piping from a readable stream to a duplex stream and just outputing that value on console.

Answer 1

I had a very similar problem. What turned out to have happened was that I was trying to pipe an object mode stream returned by MongoClient into a string/buffer stream. Which causes the error.

Judging by the snippet below:

var cursor = col.find({});

cursor.pipe(through2(function (buf, _, next) {
  console.log('chunkString: ', buf.toString());
  next();
}));

Your consuming stream is expecting a buffer.

cursor.pipe(through2({ objectMode: true }, function(chunk, enc, next) {
  console.log('chunk: ', chunk);
  next();
}));

Should solve your problem.

source: https://nodesource.com/blog/understanding-object-streams