MongoDB复制和更新同一集合中的多个文档

Question

I have some 40000 records For which I need to update a planId field and copy the records to same collection. Right Now I am using the basic

linkModel.find({},callback) & linkModel.insertMany([linksArray],callback)

But it takes up a lot of time. I have tried the aggregation but $out operator replaces the existing collection whereas I want to update it not replace.

The three steps I have tried so far:

db.links.aggregate([ { $match: { planId: ObjectId("5732f50a6886e5421259e823") } }, { $out: "temp" } ]);

db.temp.updateMany({planId:ObjectId("5732f50a6886e5421259e823")},{$set:{planId:ObjectId("5791b25f203b5e92316e60c3")}})

db.temp.copyTo("links") ,

But even copyTo method is slow. Is there a faster approach for this?

Answer 1

Take advantage of using the Bulk API for doing the updates in bulk. They offer better performance as you will be sending the operations to the server in batches of say 1000, which is much more efficient as you are not sending every request to the server, but just once in every 1000 requests.

The following demonstrates this approach, the first example uses the Bulk API available in MongoDB versions >= 2.6 and < 3.2 . It updates all the documents in the collection by changing all the Cost fields to floating value fields:

var bulk = db.links.initializeUnorderedBulkOp(),
    counter = 0;

db.links.find({ planId: ObjectId("5732f50a6886e5421259e823") })
        .snapshot().forEach(function (doc) {        
    bulk.find({ "_id": doc._id }).updateOne({ 
        "$set": { "planId": ObjectId("5791b25f203b5e92316e60c3") }
    });

    counter++;
    if (counter % 1000 == 0) {
        bulk.execute(); // Execute per 1000 operations 
        // re-initialize every 1000 update statements
        bulk = db.links.initializeUnorderedBulkOp(); 
    }
})
// Clean up remaining operations in queue
if (counter % 1000 != 0) { bulk.execute(); }

---

The next example applies to the new MongoDB version 3.2 which has since deprecated the Bulk API and provided a newer set of apis using bulkWrite() .

It uses the same cursors as above but creates the arrays with the bulk operations using the same forEach() cursor method to push each bulk write document to the array. Because write commands can accept no more than 1000 operations, you will need to group your operations to have at most 1000 operations and re-intialise the array when loop hit the 1000 iteration:

var cursor = db.links.find({ planId: ObjectId("5732f50a6886e5421259e823") }).snapshot(),
    bulkUpdateOps = [];

cursor.forEach(function(doc){        
    bulkUpdateOps.push({ 
        "updateOne": {
            "filter": { "_id": doc._id },
            "update": { "$set": { "planId": ObjectId("5791b25f203b5e92316e60c3") } }
         }
    });

    if (bulkUpdateOps.length == 1000) {
        db.links.bulkWrite(bulkUpdateOps);
        bulkUpdateOps = [];
    }
});         

if (bulkUpdateOps.length > 0) { db.links.bulkWrite(bulkUpdateOps); }