更新速度非常慢

Question

I am parsing a CSV file, for each row I want to check if corresponding entry exists in the database, and if it does I want to update it, if it doesn't I want to enter a new entry.

It is very slow - only around 30 entries per second.

Am I doing something incorrectly?

Using node, mongodb, monk

 function loadShopsCSV(ShopsName) {
    var filename = 'test.csv'

        csv
            .fromPath(filename)
            .on("data", function(data) {               
                           var entry = {
                                PeriodEST: Date.parse(data[0]),
                                TextDate: textDateM,
                                ShopId: parseInt(data[1]),
                                ShopName: data[2],
                                State: data[3],
                                AreaUS: parseInt(data[4]),
                                AreaUSX: AreaUSArray[stateArray.indexOf(data[3])],
                                ProductClass: data[5],
                                Type: data[6],
                                SumNetVolume: parseInt(data[7]),                                
                                Weekday: weekdayNum,                              
                                WeightedAvgPrice: parseFloat(data[8]),

                            }


                            db.get(ShopsDBname).update(
                                {"PeriodEST" : entry.PeriodEST,
                                 "ShopName": entry.ShopName,
                                 "State" : entry.State,
                                 "AreaUS" : entry.AreaUS,
                                 "ProductClass" : entry.ProductClass,
                                 "Type" : entry.Type},
                                  {$set : entry},
                                  function(err, result) {

                                  }
                            );
                    }
                }
            })
            .on("end", function() {
                console.log('finished loading: '+ShopsName)
            });
    }, function(err) {
        console.error(err);
    });
}

Answer 1

First I would suggest to localize problem:

• replace .on("data", function(data) with dummy .on("data", function() {return;}) and confirm speed of csv parsing.
• turn on mongo profiler db.setProfilingLevel(1) and check slow log if there is any query slower than 100 ms.

If there are no problems above - the bottleneck is in one of nodejs libraries you are using to prepare and send query.

Assuming the problem is with slow mongodb queries, you can use explain for the update query for details. It may be the case it does not use any indexes and run a table scan for every update.

Finally, it is recommended to use bulk operations, which was designed for exactly your usecase.

Answer 2

Have you tried updating with no write concern? as MongoDB blocks until whole update is successful and DB sends back that acknowledgement? Are you on cluster or something? (might want to write into primary node if so)

after your {$set : entry}, {writeConcern: {w: 0}}