Migrating data from MongoDB to SQL Server, performance degrades over time

Question

I want transfer data from MongoDB to SQL Sever using the C# driver. The data size is about 20,000,000 in three collections. I'm using 3 thread to read it and 3 threads to insert into 3 different SQL Server tables.

After reading about 70,000,000 documents, the performance becomes worse, the speed about one tenth of before, and network bandwidth slumps.

How can I prevent performance from dropping off?

Answer 1

Try migrating each collection sequentially on a single thread. The operation will be IO bound rather than CPU bound, so running it multi-threaded will more likely degrade performance and increase resource contention and context switching in the databases.

Also take a look at your SQL Server database disk writes. It might be that the hard disk cache is being used but then writes slow down as the cache is flushed to disk.