Bulk mongodb insert in Meteor or Node

Question

MongoDB supports bulk insert http://docs.mongodb.org/manual/core/bulk-inserts/

I have tried it in Meteor collection:

Orders.insert([
    { "cust_id" : "A123", "amount" : 500, "status" : "A", "_id" : "iZXL7ewBDALpic8Fj" },
    { "cust_id" : "A123", "amount" : 250, "status" : "A", "_id" : "zNrdBAxxeNZQ2yrL9" },
    { "cust_id" : "B212", "amount" : 200, "status" : "A", "_id" : "vev3pjJ8cmiDHHxe4" },
    { "cust_id" : "A123", "amount" : 300, "status" : "D", "_id" : "BBLngRhS76DgeHJQJ" }
]);

but it creates just

{ "0" : { "cust_id" : "A123", "amount" : 500, "status" : "A", "_id" : "iZXL7ewBDALpic8Fj"},
"1" : { "cust_id" : "A123", "amount" : 250, "status" : "A", "_id" : "zNrdBAxxeNZQ2yrL9" }, 
"2" : { "cust_id" : "B212", "amount" : 200, "status" : "A", "_id" : "vev3pjJ8cmiDHHxe4" }, 
"3" : { "cust_id" : "A123", "amount" : 300, "status" : "D", "_id" : "BBLngRhS76DgeHJQJ" }, 
"_id" : "6zWayeGtQCdfS65Tz" }

I need it for performance testing purposes. I need to fill and test database with thousands of testing items. I do inserts in foreach, but it takes too long to fill database.

Is here any workaround? Or can we expect Meteor will support this in next versions?

Answer 1

You could use exec ( nodejs docs ) to run a mongo script inside of meteor inside of a Meteor.startup on the server.

Example:

Meteor.startup(function () {
    var exec = Npm.require('child_process').exec;
    exec('mongo localhost:27017/meteor path-to/my-insert-script.js', function ( ) {
       // done
    });        
});

Not optimum, but I think it's your best bet for now. You can also use the command line option --eval against Mongo in exec and pass the insert statement as a string to exec. That might look like this:

Meteor.startup(function () {
    var exec = Npm.require('child_process').exec;
    exec('mongo localhost:27017/meteor --eval \'db.Orders.insert(' + JSON.stringify(arrOfOrders) + ')\'', function ( ) {
       // done
    });        
});

Answer 2

When inserting a lot of data into the DB, eg, in a forEach loop, you want to make sure that there is no reactive content on your page that depends on it. Otherwise the reactive rerendering is going to slow your client down tremendously. You can easily insert several thousand document into a collection in a fraction of a second when all templates are disabled, while the same operation can take several minutes with your CPU at 100% on both the client as the server if there is relevant reactivity happening.

You may want to add a condition to any template whose content depend on this data such as:

Template.myTemplate.items = function() {
  if (Session.get("active")) {
      return Order.find();
  }
}

Then you can deactivate all reactive rerendering before your forEach loop and reactivate it again afterwards (Session.set("active", false)).

Answer 3

You could use rawCollection which is node mongodb driver implemetation in Meteor.Collection.

await Orders.rawCollection().insertMany(arrOfOrders)

It works on 70M data in my case. (await makes it synced so you should consider to use it or not for your purpose.)