MongoDB degrading write performance over time

Question

I am importing a lot of data (18GB, 3 million documents) over time, almost all the data are indexed, so there are lots of indexing going on. The system consist of a single client (single process on a separate machine) establishing a single connection (using pymongo) and doing insertMany in batch of 1000 docs.

MongoDB setup:

• single instance,
• journaling enabled,
• WiredTiger with default cache,
• RHEL 7,
• version 4.2.1
• 192GB RAM, 16 CPUs
• 1.5 TB SSD,
• cloud machine.

When I start the server (after full reboot) and insert the collection, it takes 1.5 hours. If the server run for a while inserting some other data (from a single client), it finishes to insert the data, I delete the collection and run the same data to insert - it takes 6 hours to insert it (there is still sufficient disk more than 60%, nothing else making connections to the db). It feels like the server performance degrades over time, may be OS specific. Any similar experience, ideas?

Answer 1

I had faced similar issue, the problem was RAM. After full restart the server had all RAM free, but after insertions the RAM was full. Deletion of collection and insertion same data again might take time as some RAM was still utilised and less was free for mongo. Try freeing up RAM and cache after you drop the collection, and check if same behaviour persists.

Answer 2

As you haven't provided any specific details, I would recommend you enable profiling; this will allow you to examine performance bottlenecks. At the mongo shell run:

db.setProfilingLevel(2)

Then run:

db.system.profile.find( { "millis": { "$gt": 10 } }, { "millis": 1, "command": 1 }) // find operations over 10 milliseconds

Once done set reset the profiling mode:

db.setProfilingLevel(0)