[英]How to make multiple paginated GET requests to an API in while loop with Node.js when you don't know the page count?
I am working with a REST JSON API which provides no way to query what is the total number of pages or entries. 我正在使用REST JSON API,该API无法提供查询页面或条目总数的方法。 But I need to make multiple requests to the API in a loop in order to get all the data available.
但是我需要在一个循环中向API发出多个请求,以获取所有可用数据。
After searching through many stackoverflow questions, the closest working solution I could find successfully makes multiple requests, but still requires you to know what the last page is: 在搜索了许多stackoverflow问题之后,我可以成功找到的最接近的可行解决方案提出了多个请求,但仍然需要您知道最后一页是什么:
This works: 这有效:
const async = require("async");
const request = require("request");
let page = 1;
let last_page = 20;
let json;
async.whilst(function () {
// condition here
return page <= last_page
},
function (next) {
request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page}&per_page=20`, function (error, response, body) {
if (!error && response.statusCode == 200) {
json = JSON.parse(body);
console.log(json.data);
console.log(page);
}
page++;
next();
});
},
function (err) {
// All things are done!
});
I have tried to adapt it slightly to fit my requirements of not knowing the last page (below), but I can't figure out how to get the logic right or how to solve the asynchronous problem of getting the json
variable as undefined. 我已尝试对其稍作调整,以适应我不知道最后一页(如下)的要求,但是我无法弄清楚如何正确设置逻辑或如何解决将
json
变量定义为未定义的异步问题。 I need to get the value of the json.data array to determine the length of the data array which contains all the object data from the API response. 我需要获取json.data数组的值来确定包含API响应中所有对象数据的数据数组的长度。
This doesn't work - returns undefined
: 这不起作用-返回
undefined
:
const async = require("async");
const request = require("request");
let page = 1;
let json;
async.whilst(function () {
// condition here
json.data.length !== 0
},
function (next) {
request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page}&per_page=20`, function (error, response, body) {
if (!error && response.statusCode == 200) {
json = JSON.parse(body);
console.log(json.data);
console.log(page);
}
page++;
next();
});
},
function (err) {
// All things are done!
});
I've been working on this problem for quite a long time now, so any help would be greatly appreciated! 我已经在这个问题上研究了很长时间了,因此我们将不胜感激! Thanks!
谢谢!
There are a few problems with your logic. 您的逻辑存在一些问题。 First, you are not returning the validation in your test function on the whilst.
首先,您不会同时在测试函数中返回验证。 But even if you did, then you are testing against an undefined variable.
但是,即使您这样做了,也要针对未定义的变量进行测试。 So that validation will fail and exit your code before the first iteration.
因此,验证将失败并在第一次迭代之前退出您的代码。
let oldPage = 1;
let nextPage = 2;
let json;
async.whilst(function () {
// Check that oldPage is less than newPage
return oldPage < nextPage;
},
function (next) {
request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${oldPage}&per_page=20`, function (error, response, body) {
if (!error && response.statusCode == 200) {
json = JSON.parse(body);
console.log(json.data);
console.log(oldPage);
}
if (json.data.length) {
// When the json has no more data loaded, nextPage will stop
// incrementing hence become equal to oldPage and return
// false in the test function.
nextPage++;
}
oldPage++;
next();
});
},
function (err) {
// All things are done!
});
This way you can see the moment you don't have a new page to show anymore. 这样,您就可以看到没有新页面可以显示的时刻。
As I said, I'm not a Node.js user so not familiar with the libraries you're using, but this is at least a general idea... 就像我说的那样,我不是Node.js用户,所以对您正在使用的库不熟悉,但这至少是一个总的想法...
function getPage(page) {
request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page}&per_page=20`, function (error, response, body) {
if (!error && response.statusCode == 200) {
// do whatever you do with the current page here.
if (json.data.length == 20) {
request(`https://driftrock-dev-test-2.herokuapp.com/purchases?${page + 1}&per_page=20`, function (error, response, body) {
var json = JSON.parse(body);
if (json.data.length == 0) {
// the next page has no items - disable the next button here
}
});
}
else {
// this page has < 20 items - disable the next button here
}
}
});
}
You call that method and it will get the page specified by the parameter, and also check to see if it's the last page and disable the next button (once you add the code to do that bit). 您调用该方法,它将获得该参数指定的页面,并检查是否为最后一页并禁用下一步按钮(一旦添加代码即可完成此操作)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.