How to insert an array multiple times in mongoDB using the Node.js driver?

Question

I am trying to insert multiple objects that are contained in any given array using the mongodb-node.js driver. My intention is to modify the content of each object and immediately save those objects in a database. The following code illustrates what I want to do.

var MongoClient = require("mongodb").MongoClient;
var URL = "mongodb://localhost:27017/database";

MongoClient.connect(URL, function(err,db) {
  if(err) console.log(err);

  var array = []; // an array that serves as a data source

  for(var i=0; i<10; i++) {
    array.push({ key: i });
  };

  db.collection("collection_name", function(err,col) {
    if(err) console.log(err);

    // --- "document insertion" --- //
    array.forEach(function(doc) {
      col.insert(doc);
    });
    // --- "document insertion" --- //

  });
});

This works completely fine, as each object within the array is inserted in the database. However, I would like to perform this operation more than once; in other words, I am trying to run the "document insertion" code inside a for-loop or a while-loop where the data is modified by some function in every iteration to then be saved in mongoDB, but this simply does not work.

I have tried the following with no luck since it is as though the for-loop and the while-loop had only been executed once.

// --- "document insertion" --- //
for(var j=0; j<n; j++) { /* where n > 1 */
  /* someFunction() might be executed to alter the content of the objects */
  array.forEach(function(doc) {
    col.insert(doc);
  });
};
// --- "document insertion" --- //

This does not work either.

// --- "document insertion" --- //
while(--n) { /* where n > 1 */
  /* someFunction() */
  array.forEach(function(doc) {
    col.insert(doc);
  });
};
// --- "document insertion" --- //

I would like to perform this multiple times for large arrays (larger than 10k documents) containing more complex schema designs. The for or while loop is trying to simulate the pace of time , and each object inside the array is supposed to change as time passes by . My objective is to save every time step for all the objects in the array. How could I achieve this given that the "document insertion" piece of code seems to be performed only once? I hope I have expressed myself clearly.

Any comment is well received and appreciated. Thanks for your help.

Answer 1

The problem is that the mongoDB Node.js driver will add an _id property before inserting the document (as briefly mentioned here ), ie your array elements will have an _id property after the first loop. In the second run of the loop, this will result in duplicate key exceptions and therefore the documents are only inserted once. You can see the issue yourself when adding a callback to the insert and write out an error like so:

...
col.insert(doc, function(err) {
    if (err) {
        console.log(ErrorEvent);
    }
});
...

errmsg: 'E11000 duplicate key error collection: ...

A quick fix for this could look like this:

...
array.forEach(function (doc) {
    col.insert(doc);
    delete doc._id;
});
...

Answer 2

after manipulate your array use isertMnay() for insertion array of documents

https://docs.mongodb.com/manual/tutorial/insert-documents/#write-op-insertmany

db.collection("collection_name", function(err,col) {
if(err) console.log(err);

// --- "document insertion" --- //

  col.insertMany(doc);

// --- "document insertion" --- //

  });
  });

Answer 3

As you mentioned for 10k documents using for loop or while loop is not really good idea. It will make your node hang at some point. If you use async module, it may handle the loop.

Example:

Say for thousand times

async.times(1000, function(n, next) {
    array.forEach(function (doc) {
        col.insert(doc);
        next(err, doc);
    });
}, function(err, docs) {
    // we should now have 1000 doc. If you want to use. :)
});

This will run in parallel. If you want to make it series use timesSeries instead of times