[英]How do I manage a large number of HTTP requests in node
Been looking around the net for an answer to this, but not found anything conclusive. 一直在网上寻找答案,但没有发现任何结论。
I have a node application that (potentially) needs to make a large number of HTTP GET requests. 我有一个节点应用程序,(可能)需要发出大量HTTP GET请求。
Let's say http://foo.com/bar allows an 'id' query parameter, and I have a large number of IDs to process (~1k), ie 假设http://foo.com/bar允许使用“ id”查询参数,并且我要处理大量的ID(〜1k),即
http://foo.com/bar?id=100 http://foo.com/bar?id=100
http://foo.com/bar?id=101 http://foo.com/bar?id=101
etc. 等等
What libraries that folks have used might be best suited to this task? 人们使用了哪些图书馆可能最适合此任务?
I guess I'm looking for something between a queue and a connection pool: 我想我正在队列和连接池之间寻找某些东西:
The setup: 设置:
The process: 过程:
X
number of 'workers' is defined X
个“工人”的池 X
concurrent workers running at a time) X
并发工作人员) Any experience welcome 任何经验欢迎
It was actually a lot simpler than I initially thought, and only requires Bluebird (I'm paraphrasing here a little bit since my final code ended up much more complex): 实际上,它比我最初想象的要简单得多,并且只需要Bluebird(我在这里解释一下,因为我的最终代码最终变得更加复杂):
var Promise = require('bluebird');
...
var allResults = [];
...
Promise.map(idList, (id) => {
// For each ID in idList, make a HTTP call
return http.get( ... url: 'http://foo.com/bar?id=' + id ... )
.then((httpResposne) => {
return allResults.push(httpResposne);
})
.catch((err) => {
var errMsg = 'ERROR: [' + err + ']';
console.log(errMsg + (err.stack ? '\n' + err.stack : ''));
});
}, { concurrency: 10 }) // Max of 10 concurrent HTTP calls at once
.then(() => {
// All requests are now complete, return all results
return res.json(allResults);
});
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.