如何递归地从分页的 API 中获取数据然后组合成一个数组

Question

下面我尝试编写一个条件，它将获取分页的 api 然后 map 到另一个正在获取的 api。 即将出现的一个问题是，在拉出一个分页页面或下一页后，它不会继续循环。 第二个问题是从页面中获取的数据没有合并到一个数组中。 我做错了什么或错过了什么？

const fetchURL = `${baseURL}?owner=${accounts[0]}`;
  fetch(fetchURL, {
   method: 'GET',
   redirect: 'follow',
  })
    .then(resp => resp.json())
    .then(data => {
      console.log(data);
      const pageKey = data.pageKey
      if (pageKey !== 0) {
        fetch(`${baseURL}?owner=${accounts[0]}&pageKey=${pageKey}`, {
            method: 'GET',
            redirect: 'follow',
            })
              .then(resp => resp.json())
              .then(data => {
                console.log(data)
              })
           return data.ownedNfts.concat(data.ownedNfts)
      } else {
           return data
      }

   const responses = data.ownedNfts.map((ownedNfts) =>
       fetch(`${baseURL1}stats?address=${ownedNfts.contract.address}`)
        .then((res) => res.json()),
   );

Answer 1

要从 api 管理分页，您可以尝试这样的递归。

你应该有一个带有增量参数的请求循环的脚本，以及一个打破循环的阈值。 您必须通过时间睡眠或类似的方式来管理来自您的 api 的请求延迟。

下面的示例在带有 axios 的节点环境中工作，您可以尝试它并使其适应您的环境。

const { default: axios } = require('axios');

// Init a bigData array to push new data on each iteration
const bigData = [];

async function fetchAllPaginateData(
    pageKey = 0 /** init by default page index 0 */,
) {
    try {
        const fetchURL = `https://api.instantwebtools.net/v1/passenger?page=${pageKey}&size=1`;
        const response = await axios.get(fetchURL);
        const { data } = response;
        const { totalPages } = data; // Your api should give you a total page count, result or something to setup your iteration

        bigData.push(data); // push on big data response data

        // if current page isn't the last, call the fetch feature again, with page + 1
        if (
            pageKey < totalPages &&
            pageKey < 10 // (this is a test dev condition to limit for 10 result) */
        ) {
            pageKey++;
            await new Promise((resolve) => setTimeout(resolve, 200)); // setup a sleep depend your api request/second requirement.
            console.debug(pageKey, '/', totalPages);
            return await fetchAllPaginateData(pageKey);
        }

        console.clear();
        return console.info('Data complete.');
    } catch (err) {
        console.error(err);
    }
}

fetchAllPaginateData().then(() => console.table(bigData));

Answer 2

我稍微修改了之前的答案，使其包含在一个方法中，而不是修改较大的 scope 中的有效负载。

const axios = require('axios');
// These outer brackets are "simulating" global variables
//// `API_URL` would probably be configured globally on your request module
//// `LIMIT` might be set based on the current environment 
((API_URL, LIMIT = 10) => {
    // This method could take any structure of parameters
    // Placing them in an object may actually be better at this point
    // Defaults to an empty `combineData` array
    const fetchAllPaginateData = async (accountId, reqSize = 20, currentPage = 0, combineData = []) => {
        try {
            const fetchURL = `${API_URL}/passenger?owner=${accountId}&size=${reqSize}&page=${currentPage}`;
            const response = await axios.get(fetchURL);

            const { totalPages, totalPassengers, data } = response;
            console.debug(`${totalPassengers} Passengers`, '/', `Needing ${totalPages} pages`, '/', `${reqSize} passengers per page`);

            // Here's the secret sauce,
            // Combine data on each recursion and return it at the end
            combineData = combineData.concat(data);

            if (currentPage < totalPages && currentPage < LIMIT) {
                currentPage++;
                console.debug(currentPage, 'of', totalPages, 'pages');

                // Slow down requests so as not to DOS the API
                await new Promise((resolve) => setTimeout(resolve, 200));

                return await fetchAllPaginateData(reqSize, currentPage, combineData);
            }

            // Send big data back
            return combineData;
        } catch (err) {
            console.error(err);
        }
    }

    // Presumably this would be called from other methods or files 
    // providing the owners account number for the request query
    fetchAllPaginateData(1007).then(data => console.table(data));
})('https://api.instantwebtools.net/v1', 2);