Bulk insert and ignore if a document already exist

Question

I have a script that scraps a website everyday at a specific hour. I convert the data received into an array of objects like this:

Items = [ 
    {
      title: "some title", 
      url: "someurl.com/something" , 
      description: "some description"
     },
     {...} ,
     {...}
]

Then I use a foreach loop to insert each object of the array into MongoDB:

Items.forEach( item => Collection.create(item));

Now every day when the script runs, it gets all the previous data that have been already inserted into the database and new ones. I want to insert only new objects into the database. I tried different ways but it does not work.

Please someone can tell me how I can get it to work? Also the forEach loop I use to insert to the database I am sure there is better way to do it.

Answer 1

Use Update with upsert enabled instead of just inserting into the database.

var your_document = { .... };

Model.update(your_document, your_document, options, callback)
// is sent as
Model.update(query, { $set: ......}, options, callback)