Axios API call in nested loops doesn't return expected results

Question

I'm using Axios to scrape JSON product data from a website. The script below first requests category data from server and loops through three hierarchies of categories, subcategories, and (I guess) sub-subcategories. In the third loop, the sub-subcategories query parameter is used as a parameter in a function call, which calls an asynchronous function which uses this query parameter to concatenate onto the URL to make the new URL, make a GET request for the product data related to the query parameter, and loop this until the the pageCounter iteration variable (which is used as a pagination query parameter to get N pages for each new URL) is equal to the pagination value within the current new url containing the sub-subcategory query parameter.

    for (let i=0; i<obj.length; i++) {
        let subcats = obj[i].subcategories.length;

        for(let n=0; n<subcats; n++) {
            if(obj[i].subcategories[n].facetValueData) {
                let subsubcats = obj[i].subcategories[n].facetValueData.length

                for(let p=0; p<subsubcats; p++) {
                    let productData = []
                    obj[i].subcategories[n].facetValueData[p].productData = productData
                    const scrapedData = await scrapeData(obj[i].subcategories[n].facetValueData.code)
                    obj[i].subcategories[n].facetValueData[p].productData.push(scrapedData)
                }
            } else {
                console.log(`No facet values present - ${obj[i].subcategories[n]}`)
            }
        }
    }

This is the async function that the third loop calls

async function scrapeData(queryParam) {

    var product = []
    var pagination;

      try {
          do {
            var nextPageLink = `currentPage=${pageCounter}&query=::cagCollectionPoint:Departure+from+Singapore:cagCategory:${queryParam}`
            var nextUrl = url.concat(nextPageLink)

            const response = await axios({
                method: "GET",
                url: nextUrl, 
                withCredentials: true,
                headers: headers
            })
            product = response.data["products"]
            pagination = response.data["pagination"]["totalPages"]
            pageCounter++;

            //this logs all of the correct current queries
            console.log(response.data.currentQuery.query.value)

        } while (pageCounter<=pagination)

        return product

      } catch (error) {
          console.error(error)
      }
  }

At first glance it appears as if it is working, as it populates the first few sub-subcategory objects with an array of products scraped, however some of these are not populated, ie productData = []

The console log function in the scrapeData function returns all of the correct current queries per iteration, however when they are returned it only returns the first few reponses.

I'm guessing the product array needs to wait? But the axios request is already awaited, so i dont see why this is happening.

Answer 1

If I'm understanding your intent on this piece of code correctly:

var product = [];
do {
    // ...
    const response = await axios(/* ... */);
    product = response.data["products"]
    // ...
} while (pageCounter<=1)
return product

it seems like there are multiple pages, and you want to get all of them into the product array?

But in reality, each page you are replacing the product array. I would assume that on the empty ones your last fetch gets no results, and you are just losing all of the rest of them.

What you probably want to do is just change the product = response.data["products"] to:

product.push(...response.data["products"])