When doing an upsert to MongoDb is it possible to set a field with a timestamp only if other data in the record has changed?

Question

We need to cache records for a service with a terrible API.

This service provides us with API to query for data about our employees, but does not inform us whether employees are new or have been updated. Nor can we filter our queries to them for this information.

Our proposed solution to the problems this creates for us is to periodically (eg every 15 minutes) query all our employee data and upsert it into a Mongo database. Then, when we write to the MongoDb, we would like to include an additional property which indicates whether the record is new or whether the record has any changes since the last time it was upserted (obviously not including the field we are using for the timestamp).

The idea is, instead of querying the source directly, which we can't filter by such timestamps, we would instead query our cache which would include said timestamp and use it for a filter.

(Ideally, we'd like to write this in C# using the MongoDb driver, but more important right now is whether we can do this in an upsert call or whether we'd need to load all the records into memory, do comparisons, and then add the timestamps before upserting them....)

Answer 1

There might be a way of doing that, but how efficient that is, still needs to be seen. The update command in MongoDB can take an aggregation pipeline to perform an update operation. We can use the $addFields stage of MongoDB to add a new field denoting the update status, and we can use $function to compute its value. A short example is:

db.collection.update({
  key: 1
},
[
  {
    "$addFields": {
      changed: {
        "$function": {
          lang: "js",
          "args": [
            "$$ROOT",
            {
              "key": 1,
              data: "somedata"
            }
          ],
          "body": "function(originalDoc, newDoc) { return JSON.stringify(originalDoc) !== JSON.stringify(newDoc) }"
        }
      }
    }
  }
],
{
  upsert: true
})

Here's the playground link .

Some points to consider here, are:

1. If the order of fields in the old and new versions of the doc is not the same then JSON.stringify will fail.

2. The function specified in $function will run on the server-side, so ideally it needs to be lightweight. If there is a large number of users, that will get upserted, then it may or may not act as a bottleneck.