mongoose update millions of records while extracting information

Question

We have a production database with over 5 million customer customer records, each customer document has an embedded array of licenses they have applied for. And example customer document is as follows:

{
    _id: ObjectId('...'),
    phoneNumber: 'xxxx',
    // Other customer fields
    licenses: [
        {
            _id: ObjectId('...'),
            state: 'PENDING',
            expired: false,
            createdAt: ISODate(''),
            // Other license fields
        },
        // More Licenses for this customer
    ]
}

I have been tasked with changing the state of every PENDING license applied for during the month of September to REJECTED and sending an SMS to every customer whose pending permit just got rejected.

Using the model.where(condition).countDocuments() I have found that there is over 3 million customers (not licenses) matching the aforementioned criteria. Each customer has an average of 9 licenses.

I need assistance coming up with a strategy that won't slow down the system when performing this action. Furthermore, this is around 17GB of data.

Sending SMS is fine, I can queue details for SMS service. My challenge is processing the licenses while extracting relevant information for SMS.

Answer 1

First of all you have to create an index on the collection:

db.collection.createIndex( { "licenses.state": 1 } )

Then you shoud do something like that:

model.updateMany({}, {
    '$set': {
        'licenses.$[elem].state': 'REJECTED'
    }
}, { arrayFilters: [{
        'elem.createdAt': { $gte: ISODate(....) }
    }],
    multi: true
} ).then(function (doc)){}

If you have a replica set and your updates are on the primary instance you should not affect the secondary instances when reading on those once.

If you want to split the update on many batches you can use the _id (already indexed). Of course it depends on your _id format.