In Meteor - how to do a bulk insert into a nested array

Question

I am trying to import several thousand records into a nested array in a collection in Meteor. This is financial data coming in a JSON object. I need to do some calculations on it before insertings it, so can't do that raw. Doing a $addToSet operation for every write is very, very slow. Is there a way to push the full set of data in one call?

My schema looks something like this.

NestedSchema = new SimpleSchema({
  item: {
    type: String
  },

   aThing: {
    type: Number
   }
});

MasterSchema = new SimpleSchema({
   symbol: {
     type: String,
     label: 'Symbol',
     unique: true
   },

   data: {
     type: [NestedSchema],
     optional: true
   }
});

I have a bunch of data like this that I want to insert.

var dataToInsert = [{item: "thing", aThing: 1}, {item: "thing2", aThing: 2}, {item: "thing3", aThing: 2}];

The data I'm trying to insert into the nested array is 5000+ records. I've looked at https://atmospherejs.com/udondan/bulk-collection-update and https://atmospherejs.com/mikowals/batch-insert but they don't seem to do exactly what I'm looking for. Ideally I would have a solution where I could append new records in bulk as I collect them.

Answer 1

You can update the collection using the forEach() method on the array and within the loop take advantage of using a write commands Bulk API that allow for the execution of bulk update operations which are simply abstractions on top of the server to make it easy to build bulk operations. These bulk operations come mainly in two flavours:

• Ordered bulk operations. These operations execute all the operation in order and error out on the first write error.
• Unordered bulk operations. These operations execute all the operations in parallel and aggregates up all the errors. Unordered bulk operations do not guarantee order of execution.

Note, for older servers than 2.6 the API will downconvert the operations. However it's not possible to downconvert 100% so there might be some edge cases where it cannot correctly report the right numbers. You can get raw access to the collection and database objects in the npm MongoDB driver through rawCollection and rawDatabase methods on Mongo.Collection

MyCollection = new Meteor.Collection("mycollection");

if (Meteor.isServer) {
    Meteor.startup(function () {
        Meteor.methods({
            insertData: function(symbol) {
                var bulkOp = MyCollection.rawCollection().initializeUnorderedBulkOp(),
                    counter = 0,
                    dataToInsert = [...];

                dataToInsert.forEach(function(data) {

                    bulkOp.find({"symbol": symbol}).updateOne({ "$addToSet": data });

                    counter++;
                    if (counter % 1000 == 0) {
                        // Execute per 1000 operations and re-initialize every 1000 update statements
                        bulkOp.execute(function(e, result) {
                            // do something with result
                            console.info('result.nMatched', result.nMatched, 'result.nModified', result.nModified);
                        });
                        bulkOp = MyCollection.rawCollection().initializeUnorderedBulkOp();
                    }
                }); 

                // Clean up queues
                if (counter % 1000 != 0){
                    bulkOp.execute(function(e, result) {
                        // do something with result
                    });
                }
            }
        }); 
    });
}