Function iteration with Async waterfall in Node.js

Question

I am trying to get resource from the some url, that resource is separated as multiple pages, the number of page is over 500, but the order of resource have to be guaranteed, so I decided to use Async module.

function getData(page, callback){
    var url = "http://someurl.com/document?page="+page;
    // get data from the url ...
    // when success, 
    callback();
};

So, above function is get resource from some url, I have to iterate this function to many times, but I don't know how can I iterate this with async waterfall. What point I should push the for iteration?

async.waterfall([

    // page 1 
    function(callback){
        getData(1, function(){
            callback(null);
        });
    },

    // page 2
    function(callback){
        getData(2, function(){
            callback(null);
        });
    },

    // page 3, 4, 5..... 100, ... 500


],function(err, result){
    if(err) return next();

    console.log('finish !');
});

Answer 1

Why don't you use Promises:

function getData(page, callback){
    var url = "http://someurl.com/document?page="+page;
    // var request = call to get data from the url ... 
    return request;
};

var promises = [];

for (var page = 1; page < totalPages; page++) {
    promises.push(getData(page));
}

Promise.all(promises).then(function (listOfResults) {
    // Do callback here
});

Just make sure that your method of sending the get request returns a promise. If not you can use a function like this:

function usesCallback(data, callback) {
    // Do something with data
    callback();
}

function makePromise(data) {
    return new Promise(function (resolve, reject) {
        usesCallback(data, resolve);
    });
}

Here is some more info on Promises

Answer 2

If you do want to use async , async.map() and async.mapLimit() are better choices since they apply the iteratee to each item in parallel and guarantee the results array to be in the same order as the original collection.

async.mapLimit(_.range(1, 500), 10, getData, function (err, results) {
    // do something with results
});

The code above means to get data from page 1 to page 500, with no more than 10 async operations at a time.

_.range(1, 500) is a function from underscore to generate an array [1, 2, 3, ..., 500] . If you do not like to use underscore in your project, you can simply rewrite it such as:

function range(lower, upper) {
    return Array.apply(null, Array(upper - lower + 1))
        .map(function (_, i) { return lower + i; });
}

Answer 3

您可以使用异步的，而这在这里检查 。