如何在Express.js和Mongoose中编写异步for-each循环？

Question

I have a function that returns an array of items from MongoDB:

var getBooks = function(callback){
    Comment.distinct("doc", function(err, docs){
            callback(docs);
        }
    });
};

Now, for each of the items returned in docs, I'd like to execute another mongoose query, gather the count for specific fields, gather them all in a counts object, and finally pass that on to res.render :

getBooks(function(docs){
    var counts = {};
    docs.forEach(function(entry){
        getAllCount(entry, ...){};
    });      
});

If I put res.render after the forEach loop, it will execute before the count queries have finished. However, if I include it in the loop, it will execute for each entry. What is the proper way of doing this?

Answer 1

I'd recommend using the popular NodeJS package, async . It's far easier than doing the work/counting, and eventual error handling would be needed by another answer.

In particular, I'd suggest considering each ( reference ):

getBooks(function(docs){
    var counts = {};
    async.each(docs, function(doc, callback){
        getAllCount(entry, ...);         
        // call the `callback` with a error if one occured, or 
        // empty params if everything was OK. 
        // store the value for each doc in counts
    }, function(err) {
       // all are complete (or an error occurred)
       // you can access counts here
       res.render(...);
    });      
});

or you could use map ( reference ):

getBooks(function(docs){
    async.map(docs, function(doc, transformed){
        getAllCount(entry, ...);         
        // call transformed(null, theCount);
        // for each document (or transformed(err); if there was an error); 
    }, function(err, results) {
       // all are complete (or an error occurred)
       // you can access results here, which contains the count value
       // returned by calling: transformed(null, ###) in the map function
       res.render(...);
    });      
});

If there are too many simultaneous requests, you could use the mapLimit or eachLimit function to limit the amount of simultaneous asynchronous mongoose requests.

Answer 2

forEach probably isn't your best bet here, unless you want all of your calls to getAllCount happening in parallel (maybe you do, I don't know — or for that matter, Node is still single-threaded by default, isn't it?). Instead, just keeping an index and repeating the call for each entry in docs until you're done seems better. Eg:

getBooks(function(docs){
    var counts = {},
        index = 0,
        entry;

    loop();

    function loop() {
        if (index < docs.length) {
            entry = docs[index++];
            getAllCount(entry, gotCount);
        }
        else {
            // Done, call `res.render` with the result
        }
    }
    function gotCount(count) {
        // ...store the count, it relates to `entry`...

        // And loop
        loop();
    }
});

If you want the calls to happen in parallel (or if you can rely on this working in the single thread), just remember how many are outstanding so you know when you're done:

// Assumes `docs` is not sparse
getBooks(function(docs){
    var counts = {},
        received = 0,
        outstanding;

    outstanding = docs.length;
    docs.forEach(function(entry){
        getAllCount(entry, function(count) {
            // ...store the count, note that it *doesn't* relate to `entry` as we
            // have overlapping calls...

            // Done?
            --outstanding;
            if (outstanding === 0) {
                // Yup, call `res.render` with the result
            }
        });
    });      
});

Answer 3

In fact, getAllCount on first item must callback getAllCount on second item, ...

Two way: you can use a framework, like async : https://github.com/caolan/async

Or create yourself the callback chain. It's fun to write the first time.

edit The goal is to have a mechanism that proceed like we write.

getAllCountFor(1, function(err1, result1) {
    getAllCountFor(2, function(err2, result2) {
        ...
            getAllCountFor(N, function(errN, resultN) {
                res.sender tout ca tout ca
            });
    });
});

And that's what you will construct with async, using the sequence format.