在Replicaset上批量插入MongoDB

Question

I am working on importing data form SQL to MongoDB

I have a table in SQL with 3081583 records, I started inserting those to MongoDB in a foreach loop

It started inserting at 05-06-2014 07:42:27 3081583 records and is still running.

My connection to MongoDB using C# Driver

private static MongoDatabase ConnectionOne(string dbName = "test")
{
    var connectionString = "mongodb://username$passwordx.x.x.x:27018"
        + "/admin?slaveOk=true";

    var client = new MongoClient(connectionString);

    var server = client.GetServer();

    var database = server.GetDatabase(dbName);

    return database;
}

Is there some thing I can do to improve the insert performance? Is this right way to do the bulk insert of 3081583 records or more?

I am concerned about the performance, as this is a weekly process.

Answer 1

1. You could use bulk insert by passing array of objects to insert command. You need configure ContinueOnError and define error handling strategy. But you should split whole set on chunks so that individual chuck doesn't exceed 16Mb.

2. Generally to improve insert performance you could try to limit amount of indexes on collection, use unacknowledged write concern {w:0, j:0} - ( you would have no confirmation about insert success in this case).

Answer 2

Yes. You can insert in bulk: Bulk Inserts in MongoDB

IEnumerable<WriteConcernResult> results = collection.InsertBatch(records)

That will cut on most of the round trips to the DB which should speed things up.