在循环完成nodejs请求之前执行回调

Question

I am communicating with an API to get info on an array of ids. So each id in the Array requires a request to the api, and I want to build a array of data from the responses if they match the logic. However, The callback on my function that is processing the requests is being executed before the new array is built. I tend to run into this issue a lot when dealing with large amounts of calls to an api. How can I fix this specific example, and what is the best way to approach this in the future?

var request = require('request');
var _ = require('lodash');

var siteLayouts = [1550, 1552, 1554, 1556, 1558, 1560, 1562, 1564, 1566, 1568, 1570, 1572, 1574, 1730, 1734, 1736, 1738, 1740, 1896, 1898, 1900, 1902, 1904, 1906, 1908, 1910, 1914, 1922, 1924, 1926, 1928, 1930, 1932, 1934, 1936, 1938, 1940, 1942, 1944, 1946, 1948, 1950, 1952, 1954, 1956, 1958, 1960, 1962, 1964, 1966, 1968, 1970, 1972, 1974, 1976, 1978, 1980, 1984, 1986, 1988, 1990, 1992, 1994, 1996, 1998, 2000, 2002, 2004, 2006, 2008, 2010, 2012, 2014, 2016, 2020, 2022, 2030, 2032, 2034, 2036, 2038, 2040, 2042, 2044, 2046, 2048, 2052, 2054, 2056, 2060, 2062, 2064, 2066, 2068, 2070, 2072, 2122, 2124, 2148, 2154, 2156, 2270, 2272, 2274, 2374, 2418, 2688, 2692, 2968, 3756, 4094, 5122, 5524, 7326, 7494, 8704, 8886, 9226, 9232, 9234, 9236, 9238, 9830, 9836, 10052, 10054, 10056, 10999, 11083, 11085, 11429, 11513, 17279, 20397, 22285, 22287, 22289, 22291, 22293, 22295, 22807, 22809, 22811, 22813, 22815];

function getLayoutModules(siteLayouts, callback) {
    var matchedModules = [];
    for (var i = 0; i < siteLayouts.length; i++) {
        request('http://PRIVATE-API-URL/layout/' + siteLayouts[i], function(err, res, body) {
            if (!err && res.statusCode == 200) {
                var layoutModules = JSON.parse(body);
                var match = _.filter(layoutModules, {
                    'dtoLayoutModule': {
                        'ModuleName': 'Featured Content'
                    }
                });
                if (match.length > 0 && match[0].dtoLayoutModule) {
                    //console.log(match[0].dtoLayoutModule);
                    matchedModules.push(match[0].dtoLayoutModule);
                    console.log(matchedModules.length)
                }
            }
        });
    }
    callback(matchedModules);
}

getLayoutModules(siteLayouts, function(matchedModules) {
    console.log(matchedModules);
});

I have verified the data is being added to the final array via the console.log length but I see the callback console.log first then the lengths. Also here is an example response body from the request after it is filtered with _.filter

[{
    RequestStatus: {
        StatusCode: '200',
        StatusTxt: 'Successful request.',
        Result: 'Successful request.',
        ValidationErrors: null
    },
    dtoLayoutModule: {
        Id: 116013,
        LayoutId: 10999,
        LayoutName: 'layout name',
        ModuleId: 7,
        ModuleName: 'Featured Content',
        DisplayName: 'name to display',
        Position: 4,
        Config: '<config><item name="layout" value="primary" /></config>',
        MaxContentCount: 4,
        CanInherit: false,
        IsInheritable: false,
        IsStaticModule: false
    }
}]

Answer 1

for loops don't work as you might hope with asynchronous code as they are intended for use with synchronous code only. Use async.map to handle the async requests for each ID. That will take your array of IDs, do an async request for each one, and give you back an array of responses, which you can then filter with regular Array.prototype.filter before finally passing that matchedModules to your callback.

Answer 2

The problem you are heaving is you are thinking synchronously, and are working asynchronously. You are firing your requests in a loop, and invoke your callback right after the loop. But you have no guaranty that all the calls will have been processed by the time you get to the invocation of the callback method. Actually the opposite is pretty mush guarantied. What you need is a way to invoke your callback only after all requests have been processed.

I think you should consider using promises. Promises are a part of Ecmascript6, and are used intensivly today via various libraries. The most famous one (as far as I know) is q, but there are others.

The promise pattern tries (and does a pretty good job) to eliminated what is referred to as 'callback hell'.

Here's an example of the pattern:

asyncCall()
  .then(function(data1){
     // do something...
     return anotherAsyncCall();
   })
   .then(function(data2){
     // do something...  
     return oneMoreAsyncCall();    
   })
   .then(function(data3){
      // the third and final async response
   })
   .fail(function(err) {
      // handle any error resulting from any of the above calls    
   })
   .done();

This example was taken from this article called Promises – an alternative way to approach asynchronous JavaScript.

In your particular case, one way to go about it, is to collect the promises (each promise is created for each request) into an array of promises, then use Q.all method. That promise is resolved when all promises have been resolved.