MongoDB very slow read performance on server

Question

Basically, I have a Windows service which performs a batch job.

I have two collections that are related, customerAccounts and events. The events collection logs actions that customers performed on a site, containing the timestamp, the name of the event, the page it occurred on and the username.

The service runs through each account and works out their journey phase and risk of account closure based on what events they have in the Events collection and a set of user-defined rules.

There are about 3,500 accounts and around 100,000 events in my database at present. The service takes just over 1 minute to run on my development PC, but takes seemingly forever on the server (I've estimated it takes roughly 2.5 hours based on modifying the service so it only performs the job on a single customer account.

My machine is a Core i7 with 16GB of RAM, the server is an Intel Xeon E5-2609 (64bit, Win 2008 R2) with 24GB of RAM. I put the database on a much older server (32bit, Windows 2003) and the service took about 2 minutes to run. So, I know that on my dev machine it takes just over a minute, and on older server hardware it takes just over 2 minutes, yet on a modern server it's taking a matter of hours.

Originally, the Mongo Shell was warning that NUMA was enabled on the server and should be switched off to avoid performance problems. This has since been turned off but hasn't seemed to have an affect on performance.

When I run db.currentOp(); on the server, I've noticed that it's always got a query "createIndexes" of some nature (the indexes were created ages ago), yet when I mongodump/restore the database to my dev machine and run the service/currentOp, the "createIndexes" query isn't there. Apart from that, nothing else seems to jump out at me.

Does anyone have any ideas / help on this mysterious performance issue? I'll post currentOp/mongostats if/when required.

Answer 1

Quick answer: I re-installed Mongo. No fancy configuration, just ran the setup and it fixed the issue.

I never worked out why Mongo was constantly creating indexes. The log file is 0.25GB for a single day full of event logs for "creating index".