Best way to process results of multiple, sequential, dependent mongo queries in NodeJS

Question

High level

I'm new to JS and Node. I'm working on an API endpoint that should return a status value. To compute the status value I need to make two sequential mongo queries where the second set of queries depend on the first query. The second set of queries will give me a status for each value found in the first query, of which I will pick one based on some logic. What is the best way to do it in NodeJS?

Specifics

Here are parts of my first attempt.

function getItemStatus(key, value, callback) {
        MongoClient.connect(mongo_url, function(err, db) {
            if (err) { return console.dir(err); }
            db.collection('status', function(err, coll) {
                if (err) { return console.dir(err); }
                coll.distinct("_id.metric", function(err, metrics) {
                    if (err) { return console.dir(err); }
                    console.log('metrics : ', metrics);
                    _.foreach(metrics, function(metric) {
                        var query = {"_id": {
                            "$gte" : {key: key, value: value, created: new Date("1800-01-01T00:00:00"), metric : metric},
                            "$lte" : {key: key, value: value, created: new Date("2100-01-01T00:00:00"), metric : metric}}};
                        coll.find(query, {sort: {"_id.created": -1}, limit: 1})

I make a connection, query for a set of metric values using a distinct query. For each metric I then want to ask for the latest status. Ideally I'd like to have the entire set of statuses so that I could write a function taking this set and deciding on which status will be returned. My problem is passing the statuses back "up the chain" so that I can process the set of statuses.

In a synchronous situation I would simply write something like this

val metrics = getDistinctMetrics(key, value)
val statuses = metrics.map(getStatusForMetric)
val status = filterStatuses(statuses)

How can I accomplish this in JavaScript/NodeJS?

UPDATED to highlight the fact that the first queries will trigger several queries in the second step, ie one for each result found by the first query.

Answer 1

As I understand your question you want to execute queries parallel or in a waterfall mode and do some logic on the final result. You should look into a library allowing parallel/waterfall execution. Like this

Waterfall: Waterfall

    async.waterfall([
    function(callback) {
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback) {
        // arg1 now equals 'one' and arg2 now equals 'two'
        callback(null, 'three');
    },
    function(arg1, callback) {
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
    // result now equals 'done'
});

Parallel: Parallel

async.parallel({
    collectionOne: function (callback) {
        collectionOne.find(query, function (err, result) {
            if (err) {
                return handleError(res, err);
            }
            callback(null, result);
        })
    },
    collectionTwo: function (callback) {
        collectionTwo.find(query, function (err, result) {
            if (err) {
                return handleError(res, err);
            }
            callback(null, result);
        })
    },
    collectionThree: function (callback) {
        collectionThree.find(query, function (err, result) {
            if (err) {
                return handleError(res, err);
            }
            callback(null, result);
        })
    },
    collectionFour: function (callback) {
        collectionFour.find(query, function (err, result) {
            if (err) {
                return handleError(res, err);
            }
            callback(null, result);
        })
    },
}, function (err, results) {
    return res.status(200).json(results);
});

And in the final callback you can doo some logic or return response.

Answer 2

In your sample code, you are making network calls inside another network calls, which can lead to callback hell which can lead to misbehave of queries, in order to overcome that, you can use promises .

This will help you in avoiding callback hell as well as your query will also be resolved.

Sample code:-

new Promise (function(resolve, reject){
      return db.collection.find(function).exec()
      .then(function(result_of_first_query){
            return db.collection.findOne(function).exec()   //in this yopu can user the result of first query
      }).then(function(result_of_second_query){
             resolve(result_of_second_query);
      })
})

You can add more queries with each .then