Mongodb - Updating large number of records every minute

Question

There are hundreds of sensors which throws around 100,000 records every minute.

So I need to catch and update this data to a Nodejs + Mongodb server every minute via Api.

Now this is making the server so slow that I am unable to retrieve data for reporting purpose.

Even my server Web Url keeps loading non stop.

In order to update all those records, What I am doing on this server right now is:

for(var i = 0; i < records.length; i++) {

      SensorData.update({storeId: records[i].id}, {$set: {inTime:records[i].inTime, outTime: records[i].outTime, timestamp: records[i].createdAt}});

}

The reason this query is in loop is because it updates records on multiple attributes like stores, building, brands, shelfs etc

Can anyone help me with the best architecture possible?

Answer 1

The best architecture depends on the resources you can spare on your setup (including many other things). If you are experiencing slowness, it could be the limitation on systems resources (CPU, Mem, network, etc.). So monitor the resource usage on your MongoDB nodes.

But you can speed up your writes to MongoDB using insertMany or bulkWrite (as others have suggested too). insertMany writes multiple documents in one go and bulkWrite allows multiple operations.

For instance, you can combine document objects in an array and write to MongoDB using insertMany . Because your objects array will be stored in memory, how many documents you can write at once depends on the memory available to your application.

A simple example like,

let objArray = [];
for(var i = 0; i < records.length; i++) {
    // add each doc to array
    objArray.push({storeId: records[i].id}, {$set: {inTime:records[i].inTime, outTime: records[i].outTime, timestamp: records[i].createdAt}});
}
// using insertMany
db.collection(collectionName).insertMany(objArray, (err, res) => {
    if (err) {
        // handle error
    }
    if (res) {
        // handle result
    }
}

Edit :

As you want updates rather than inserts, you can try with updateMany ( bulkWrite also allows you to do this).

// provide filters to match docs, or an empty object to match all i.e. {}
let matchThis = { field : {$eq : “abc” } };
// update matched docs
var toUpdate = { $set : {field : “xyz” } };

// make updates
db.collection(collectionName).updateMany(matchThis , toUpdate, (err , res) =>  {
    // handle response
});

Note that available systems resources play a big part in performance, even with doing updates/writes in bulk (time spent in to and fro calls will be reduced significantly, however).