[英]Async.whilst with requests and unknown page limit - how to make requests asynchronous?
So I have written a scrip which works nicely, but I want to make the requests asynchronous. 所以我写了一个很好的脚本,但是我想使请求异步。 Currently, each one runs after the previous has finished, I had to do it this way because I have no way of knowing the total # of pages. 当前,每个页面都在上一个页面完成后运行,因此我必须这样做,因为我无法知道页面总数。 I have to hit the requests until I get an error. 我必须按要求,直到出现错误。
Is there a better way of writing this code? 有没有更好的方式编写此代码?
var getResponse = 'true'; var getResponse ='true'; var newNum = 0; var newNum = 0; var responses = []; var responses = [];
async.whilst( function () { // starts the async while loop return getResponse == 'true'; async.whilst(function(){//启动异步while循环return getResponse =='true';
},
function (callback) {
// calls everytime
newNum++;
console.log('num',newNum);
var options = {
url: 'http://upload-api.kooaba.com/api/v4/buckets/sdfsfsfsfsdfsd/items?page='+newNum,
}
request(options, function(error, response, body) {
var info = JSON.parse(body);
if(info.offset) {
getResponse = false;
console.log('end of the records');
console.log("All loaded.");
var jsonSize = _.size(responses);
console.log('size',jsonSize);
res.json(responses);
}
else {
_.each(info, function(obj) {
var imageObj = {
uuid : obj.uuid,
enabled: obj.enabled,
title: obj.title,
itemCode: obj.reference_id,
num_images: obj.num_images,
bucket: 'uk'
}
responses.push(imageObj);
});
callback();
}
});
},
function (err) {
// End loop
console.log('end');
}
); );
Async.js tends to work on known sets of data, and will not accept changes to them once initiated. Async.js倾向于处理已知的数据集,并且一旦启动就不会接受对其的更改。
Since no one else replied, I'm posing a general idea for loop based processing of the apps. 由于没有其他人回答,因此我提出了基于循环处理应用程序的总体思路。 This is a hacky way to do what you seek using a synchronous while loop, although this might need to be optimized as it may prove as CPU costly: 这是一种使用同步 while循环来完成您想要的事情的简单方法,尽管由于可能会浪费大量CPU资源,因此可能需要对其进行优化:
var flag = false; // Will be used to indicate loop break time.
var expectedResponses = 0; // Will count sent requests
var arrRequests = []; // Will be used as the queue, constantly getting requests pushed.
var arrResponses = []; // Will be used to keep the responses.
while (!flag || expectedResponses < arrResponses.length) {
if (arrRequests.length) {
var requestParams = arrRequests.pop();
expectedResponses++;
request(requestParams, function cb(err, res) {
if (err) {
// Handle error and:
expectedResponses--;
return;
}
arrResponses.push(res);
});
}
}
// Process responses here.
// This code will not be reached before flag is raised, AND queue is processed.
Explanation: This sync loop does not wait for a callback. 说明:此同步循环不等待回调。 It will iterate every time the internal code is processed, and the closing }
is reached. 每次处理内部代码并到达结束}
时,它都会迭代。 The reason being it waiting for a return statement, which is received as a return null;
原因是它等待一个return语句,该语句作为return null;
接收return null;
Note flag should be raised manually to allow breaking of the loop once all requests had received a response. 注意标志应手动提高以允许一旦所有请求都收到响应后就中断循环。 Also, you might want to consider re-writing this as event based, having new request event, and a second event which fires once flag is raised and last response is received.. 另外,您可能需要考虑将其重写为基于事件的事件,具有新的请求事件,而第二个事件将在引发标志并接收到最后一个响应时触发。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.