Node.js: serial operations with branching

Question

I'm new to node.js and using it for a backend that takes data from syslog messages and stores it to a database.

I've run into the following type of serial operations:

1. Query the database

2. If the query value is X do A. otherwise do B.
    A. 1. Store "this" in DB
       2. Query the database again
       3. If the query value is Y do P. otherwise do Q.
           P. Store "something"
           Q. Store "something else"

    B. 1. Store "that" in the DB
       2. Store "the other thing" in the DB

The gist here is that I have some operations that need to happen in order, but there is branching logic in the order.

I end up in callback hell (I didn't know what that is when I came to Node.. I do now).

I have used the async library for things that are more straight forward - like doing things in order with async.forEachOfSeries or async.queue . But I don't think there's a way to use that if things have to happen in order but there is branching.

Is there a way to handle this that doesn't lead to callback hell?

Answer 1

Any sort of complicated logic like this is really, really going to benefit from using promises for every async operation, especially when you get to handling errors, but also just for structuring the logic flow.

Since you haven't provided any actual code, I will make up an example. Suppose you have two core async operations that both return a promise: query(...) and store(...) .

Then, you could implement your above logic like this:

query(...).then(function(value) {
    if (value === X) {
        return store(value).then(function() {
            return query(...).then(function(newValue) {
               if (newValue === Y) {
                   return store("something");
               } else {
                   return store("something else");
               }
            })
        });
    } else {
        return store("that").then(function() {
            return store("the other thing");
        });
    }
}).then(function() {
    // everything succeeded here
}, function(err) {
    // error occurred in anyone of the async operations
});

I won't pretend this is simple. Implemented logic flow among seven different async operation is just going to be a bit of code no matter how you do it. But, promises can make it less painful than otherwise and massively easier to make robust error handling work and to interface this with other async operations.

The main keys of promises used here are:

1. Make all your async operations returning promises that resolve with the value or reject with the failure as the reason.
2. Then, you can attach a .then() handler to any promise to see when the async operation is finished successfully or with error.
3. The first callback to .then() is the success handler, the second callback is the error handler.
4. If you return a promise from within a .then() handler, then that promise gets "chained" to the previous one so all success and error state becomes linked and gets returned back to the original promise state. That's why you see the above code always returning the nested promises. This automates the returning of errors all the way back to the caller and allows you to return a value all the way back to the caller, even from deeply nested promises.
5. If any promise in the above code rejects, then it will stop that chain of promises up until an actual reject handler is found and propagate that error back to that reject handler. Since the only reject handler in the above code is at the to level, then all errors can be caught and seen there, no matter how deep into the nested async mess it occurred (try doing that reliably with plain callbacks - it's quite difficult).

Native Promises

Node.js has promises built in now so you can use native promises for this. But, the promise specification that node.js implements is not particularly feature rich so many folks use a promise library (I use Bluebird for all my node.js development) to get additional features. One of the more prominent features is the ability to "promisify" an existing non-promise API. This allows you to take a function that works only with a callback and create a new function that works via a promise. I find this particularly useful since many APIs that have been around awhile do not natively return promises.

Promisifying a Non-Promise Interface

Suppose you had an async operation that used a traditional callback and you wanted to "promisify it". Here's an example of how you could do that manually. Suppose you had db.query(whatToSearchFor, callback) . You can manually promisify it like this:

function queryAsync(whatToSearchFor) {
    return new Promise(function(resolve, reject) {
        db.query(whatToSearchFor, function(err, data) {
           if (!err) {
               reject(err);
           } else {
               resolve(data);
           }
        });
    });
}

Then, you can just call queryAsync(whatToSearchFor) and use the returned promise.

queryAsync("foo").then(function(data) {
    // data here
}, function(err) {
    // error here
});

Or, if you use something like the Bluebird promise library , it has a single function for promisifying any async function that communicates its result via a node.js-style callback passed as the last argument:

var queryAsync = Promise.promisify(db.query, db);