Node.js承诺递归

Question

Im having trouble with iterating through my Promises and finishing my parser code:

let startFrom = 0,
    totalRecords = 10000,
    doneRecords = 0
const rows = 10

const parserRequests = function () {
if (startFrom <= totalRecords) {
    setTimeout(function () {
        getParserOffers(startFrom)
        startFrom += rows
        parserRequests()
    }, 2000)
}}

const getParserOffers = function (start) {
request({
    uri: domain + '/lucene/search/filter?minPrice=0.00&maxPrice=639.63&stock=all&cat=0&sortOrder=added+desc&start=' + start + '&rows=' + rows + '&steam_app_id=&steam_category=&steam_prod_type=&includeOutOfStock=&includeFreeGames=false',
    json: true
}).then(response => {
    const promises = response.docs.map(doc => getOffersData(doc))
    return Promise.all(promises)
}).then(gamesList => {
    doneRecords += rows
    console.log('Done records: ', doneRecords)
    if (doneRecords >= totalRecords) {
        process.exit()
    }
}).catch(error => {
    console.error('ERROR: Server did not respond')
})}

In recursive parserRequests function I iterate till totalRecords and in getParserOffers after getting gamesList I also iterate every time, when it's done. The problem is that doneRecords should also reach 10000 number to execute process.exit() and stop the program, but it never does, it stops for example on 9750. What can be the problem? Thank you for help

Answer 1

First of all, it doesn't look like getParserOffers() method executes sequentially. It is called every 2 seconds, and if the host that you're querying can't handle the eleven HTTP requests per given time, then your requests are queued on the host and your promises stacked waiting.

I think you meant for it to execute sequentially. For that you need to return your promise:

const getParserOffers = function (start) {
  return request({

and attach to it the next iteration via then() method:

if (startFrom <= totalRecords) {
    setTimeout(function () {
        getParserOffers(startFrom).then(_ => {
            startFrom += rows
            parserRequests()
        });
    }, 2000);
}

you might also want to decrease the delay since it would now execute sequentially one after another.

I would also recommend changing this part:

doneRecords += rows
console.log('Done records: ', doneRecords)
if (doneRecords >= totalRecords) {
    process.exit()
}

move it to the place where you set the timeout, you don't have to process.exit(), you could simply stop setting the next timeout and your program would finish executing.