Node.js and MongoDB if document exact match exists, ignore insert

Question

I am maintaining a collection of unique values that has a companion collection that has instances of those values. The reason I have it that way is the companion collection has >10 million records where the unique values collection only add up to 100K and I use those values all over the place and do partial match lookups.

When I upload a csv file it is usually 10k to 500k records at a time that I insert into the companion collection. What is the best way to insert only values that dont already exist into the unique values collection?

Example:

//Insert large quantities of objects into mongo
    var bulkInsert = [
        {
            name: "Some Name",
            other: "zxy",
            properties: "abc"
        },
        {
            name: "Some Name",
            other: "zxy",
            properties: "abc"
        },
        {
            name: "Other Name",
            other: "zxy",
            properties: "abc"
        }]
 //Need to insert only values that do not already exist in mongo unique values collection   
    var uniqueValues = [
        {
            name:"Some Name"
        },
        {
            name:"Other Name"
        }
    ]

EDIT I tried creating a unique index on the field, but once it finds a duplicate in the Array of documents that I am inserting, it stops the whole process and doesnt proceed to check any values after the break.

Answer 1

Figured it out. If your doing it from the shell, you need to use Bulk() and create insert jobs like this:

var bulk = db.collection.initializeUnorderedBulkOp();
bulk.insert( { name: "1234567890a"} );
bulk.insert( { name: "1234567890b"} );
bulk.insert( { name: "1234567890"} );
bulk.execute();

and in node, the continueOnError flag works on a straight collection.insert()

collection.insert( [{name:"1234567890a"},{name:"1234567890c"}],{continueOnError:true}, function(err, doc){}

Answer 2

Well, I think the solution here is quite simple if I understand correctly your issue. Since the process is stopped when it finds a duplicated field you should basically check if the value doesn't already exists before to try to add it.

So, for each element in uniqueValues , make a find/findOne query, if it doesn't return any result then add the element, otherwise don't.