[英]Send Multiple HTTP requests
我需要编写程序来使用项目 ID 查找有关项目的信息。
API 一次只取一项,所以我每项只能执行一个查询。 API 仅限于五个同时请求。 任何额外的结果都会给出 HTTP 429 错误。
如果有一个 JavaScript Object ,其中所有项目都有他们的 ID
如何在不触发同时请求限制的情况下检索所有给定 ID 的信息,并且不对已经看到的项目 ID 执行不必要的查询。
import fetch from "node-fetch";
let itemObject = [
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
];
async function sendIDRequests() {
try {
const response = await fetch("https://url/items/:ID", {
headers: {
Authorization: "",
},
});
if (!response.ok) {
throw new Error(`${response.status} ${response.statusText}`);
}
response
.text()
.then((res) => console.log(res))
.catch((err) => {
throw new Error(err);
});
} catch (error) {
console.error(error);
}
}
sendRequests()
为此,我想到了两种方法。 批处理和滑动 window 方法。 批处理可能更容易,但使用滑动 window 将是更有效的实现。
这种方法创建的请求批次达到指定的batchSize
,并且只有在一个批次中的所有请求都完成后,才会发出下一批请求。
您需要在此处添加一些错误处理以防请求失败。
import fetch from "node-fetch";
// list of items that you might want to use to compose your URL (not actually used here)
let itemObject = [
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
];
(async () => {
// number of concurrent requests in one batch
const batchSize = 4;
// request counter
let curReq = 0;
// as long as there are items in the list continue to form batches
while (curReq < itemObject.length) {
// a batch is either limited by the batch size or it is smaller than the batch size when there are less items required
const end = itemObject.length < curReq + batchSize ? itemObject.length: curReq + batchSize;
// we know the number of concurrent request so reserve memory for this
const concurrentReq = new Array(batchSize);
// issue one request for each item in the batch
for (let index = curReq; index < end; index++) {
concurrentReq.push(fetch("https://postman-echo.com/get"))
console.log(`sending request ${curReq}...`)
curReq++;
}
// wait until all promises are done or one promise is rejected
await Promise.all(concurrentReq);
console.log(`requests ${curReq - batchSize}-${curReq} done.`)
}
})();
预期结果:
sending request 0...
sending request 1...
sending request 2...
sending request 3...
requests 0-4 done.
sending request 4...
sending request 5...
sending request 6...
sending request 7...
requests 4-8 done.
sending request 8...
sending request 9...
sending request 10...
sending request 11...
requests 8-12 done.
此方法使用滑动 window并在另一个请求完成后立即安排新请求,同时始终保持请求计数低于或等于任意一次n
请求的最大数量。 你需要实现它的是一个Semaphore 。
在 JavaScript 中有一个用于此的库,称为async-mutex 。
这里的示例程序使用此库同时向 Postman Echo API 发送 2 个请求。 在信号量允许的情况下,永远不会有更多的请求同时运行(在您的情况下,限制为 5,这里为 2)。
import { Semaphore } from "async-mutex";
import fetch from "node-fetch";
// list of items that you might want to use to compose your URL (not actually used here)
let itemObject = [
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
{ itemName: "", itemID: "" },
];
(async () => {
// allow two concurrent requests (adjust for however many are required)
const semaphore = new Semaphore(2);
itemObject.forEach(async (item, idx) => {
// acquire the semaphore
const [value, release] = await semaphore.acquire();
// at this point the semaphore has been acquired and the job needs to be done
try {
console.log(`sending request ${idx}...`)
const response = await fetch("https://postman-echo.com/get")
if(!response.ok){
console.log(`request failed with status code ${response.status}`)
}
}
catch (error) {
console.log("request failed.")
}
finally {
console.log(`request ${idx} done...`)
// release the semaphore again so a new request can be issued
release();
}
})
})();
预期 output(订单可能有所不同):
sending request 0...
sending request 1...
request 1 done...
sending request 2...
request 2 done...
sending request 3...
request 3 done...
sending request 4...
request 0 done...
sending request 5...
request 4 done...
sending request 6...
request 5 done...
sending request 7...
request 6 done...
sending request 8...
request 7 done...
sending request 9...
request 8 done...
sending request 10...
request 9 done...
sending request 11...
request 10 done...
request 11 done...
等待个别 API 调用完成
尝试await sendRequests()
- sendRequests()
() 返回的未决 promise 被丢弃,因为它没有被传递给await
运算符或已添加then
、 catch
或finally
子句。
如果您希望await sendRequests()
在由response.text()
启动的 promise 链的子句已执行,而不是简单地定义(在sendRequests
内同步发生)之后执行,请在response.text()
:
return response.text()
.then // ... rest of promise chain code
这迫使await sendRequests()
等待执行 promise 链处理。
计算未完成的请求
尝试将sendRequests
重命名为sendRequest
(单数),并编写一个节点模块(可能是sendRequests
)来记录已发出但仍在等待响应的请求的计数。 它将为单个请求返回 promise,但在未完成请求的计数低于允许的限制之前不会发出新的提取操作。
这种模块的复杂性取决于设计标准:
使用模块化工厂 function 或 class 构造函数来创建定制的sendRequests
function 的通用解决方案对于您的用例来说可能会也可能不会过度。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.