How can we monitor mongoDB queries?

Question

I just wanted to know if there is a possible way to monitor mongoDB queries. I tried with the explain functionality provided by the mongo shell. But it is very hectic to manually track each and every query. I am using mongoose as an ODM.

The one which I tried is:

db.customer.find({},{name:1, active:1}).explain()

I got an object with the query plan, time taken and many more things.

Answer 1

Mongoose out-of-the-box supports only basic debug:

mongoose.set('debug', true);

But that doesn't measure query time so is almost no use for profiling. Since mongoose 4.* you can use middleware to measure request time: http://mongoosejs.com/docs/middleware.html

There are some nodejs libs to measure execution time of different code blocks and app performance:

• http://www.devmetrics.io/logs?section=performance
• http://benchmarkjs.com
• ...

Answer 2

The number of read (query, getmore) and write (insert, delete, update) operations are reported in opcounters under the serverStatus command. Remember that you should also correlate these throughput statistics along with resource saturation metrics such as currentQueue.readers and currentQueue.writers (also part of serverStatus ).

Here are detailed all the different ways to collect the metrics you need: using Utilities, Commands, or monitoring tools integrating with MongoDB (in the same series you will also find all the statistics you need to properly monitor MongoDB).

Answer 3

What you need is debugging mode :

All executed collection methods will log output of their arguments to your console

mongoose.set('debug', true);

Or you could add callback as third argument that allows you get additional info:

mongoose.set('debug', function (collection, method, query, doc [, options]) {
   console.log(/* your log format */);
});

MongoDB also provides monitoring of your mongod server in a cloud with MMS

EDIT: to save your queries in csv you could use csv-write-stream module with the following example:

var csvWriter = require('csv-write-stream');
var fs = require('fs');
var writer = csvWriter();

// create write stream to `queries.csv` file
writer.pipe(fs.createWriteStream('queries.csv'));

mongoose.set('debug', function (collection, method, query, doc [, options]) {
   writer.write({collection: collection, method: method, query: query, doc: JSON.strinfigy(doc)});   
});

// close stream on mongoose disconnected
mongoose.connection.on('disconnected', function () {  
  writer.end(); 
});

Answer 4

Use db.setProfilingLevel() to log slow queries or all queries. Then, use ElasticSearch + Kibana + Logstash to analyze and monitor mongoDB queries.

Answer 5

I found a better way of doing this by using mongoDB profiler . It writes the profile result into a collection called system.profile which can be queried as other collections and can be exported too.

There are two ways of enabling profiler:

1. Enable profiler for all the databases at once
2. Enable profiler for a particular database.

The following profiling levels are available:

0: No profiling, 1:profile slow operations and 3:profile all operation

See: db.setProfilingLevel(2) and db.getProfilingStatus()