inserting max num of records in mongodb using nodejs in less time

Question

I'm doing a small task that, Need to read a large file(ie, 1.3GB) in node and devide every line into one record, every record to be inserted into mongodb collection in less time. Please suggest me in code and thanks in advance.

Answer 1

You probably want to read a such amount of data without buffering it into memory.

Assuming you are dealing with JSON data, I think this might be a feasible approach:

var LineByLineReader = require('line-by-line');
var fileHandler = new LineByLineReader('path/to/file', { encoding:'utf8', skipEmptyLines: true });
var entries = [];
var bulkSize = 100000; // tweak as needed

fileHandler.on('error', function (err) {
  // process errors here
});

fileHandler.on('line', function (line) {
  entries.push(JSON.parse(line));
  if (entries.length === bulkSize) {
     // pause handler and write data
     fileHandler.pause();

     YourCollection.insertMany(entries)
     .then(() => { 
       entries = [];
       fileHandler.resume();
     })
  }
});

fileHandler.on('end', function () {
  YourCollection.insertMany(entries)
  .then(() => {
    // everything's done, do your stuff here
  });
});

The line-by-line module seems to be a bit buggy and could be deprecated in the future , so you may want to use linebyline instead