Sequential execution of async calls in Node.js

Question

I'm trying to run Node.js async functions in a loop. Each function performs an API call and logs the response to the console and to a file. The problem is that the logs are written only after all the calls have finished execution, and I'd like the output to be logged sequentially.

Here is the code:

const checkBatch = async function () {
  const MAX_BATCH = 20;
  for (let i = 0; i < MAX_BATCH; i++) {
    await apiCallAndLog();
  }
};


const q = async.queue((task, cb) => { console.log(`${task.name} processed`); cb(); });
for (let i = 0; i < MAX_CHECKS; i++) {
  q.push({ name: 'batch' + i }, checkBatch);
}

Is it possible to fix this so that the output is logged sequentially and not after all the batches have finished? I've tried simple loops and the async queue (as in this sample).

Answer 1

You need to use checkBatch as the worker of the queue, not as the callback for the individual item:

const q = async.queue((task, cb) => {
  checkBatch(task).then(res => cb(null, res), err => cb(err));
});
for (let i = 0; i < MAX_CHECKS; i++) {
  q.push({ name: 'batch' + i }, (err, res) => console.log(`batch${i} processed`);}););
}

However, async.js doesn't harmonise well with promises as you can see, you'd better choose a promise-aware queuing library.

Or drop the need for a queue completely and just write

for (let i = 0; i < MAX_CHECKS; i++) {
  await checkBatch({ name: 'batch' + i });
  console.log(`batch${i} processed`);
}