比较数百万条 mongoDB 记录变化的最佳方法

Question

I am working on a project where I store dns records of millions of websites and I need to monitor and update changes in these data periodically. The data is stored on a mongodb as follows

{
  domain: "www.google.com",
  "IP": [
         {
           "value":"216.58.198.78",
           "first_seen":"2020-02-01 00:00:00",
           "last_seen":"2020-02-10 00:00:00"
          },
        
          {
           "value":"216.58.198.75",
           "first_seen":"2020-02-11 00:00:00",
           "last_seen":"2020-02-25 00:00:00"
          },
          ...
         ]
        
}

I run periodic scans to get new domains and fresh DNS records and I would like to know the best way to compare it with data stored in DB and update it.

What I am thinking, is to do the following.

1. Retrieve all records from DB (I do not think this is good at all)
2. Store retrieved data into a python dictionary with domain as its key
3. Loop through the fresh records
4. Check if the domain exists within dictionary, then compare changes and perform necessary updates to the dictionary.
5. If the domain does not exist, add it to the dictionary
6. Drop the collection?
7. Perform bulk write operation to store the new values

This sounds terrible in performance and memory consumption (we are storing millions of records in memory) but I am not sure if other alternatives (query then update) would do any better (cuz we'd need to perform millions of transactions)

I would appreciate if you can provide some insights on the best way to achieve this or guide me to areas of research that might help.

Thanks

Answer 1

The normal practice is to add a data field (eg "NeedUpdate") on the database table.

On creating a new record, the "NeedUpdate" will be "ON" for that record

Upon updating an existing record, the "NeedUpdate" will be set as "ON" too

After that, you can run a cron job (or any period scans) to process the records with "NeedUpdate"="ON" (and after processing, set the "NeedUpdate=''".

In that case the system only needs to process the records which require update.