Parallel requests with map and for loop with promise.all and then

Question

I am trying to run parallel requests to a third party API for hundreds of keywords and with five different types of request on each, it's working but after the promises resolve I have to manipulating the data further and sometimes it comes a bit earlier.

const myFunc = async () => {

  const keywords = ['many', 'different', 'type', 'of', 'keywords']

  // Create promise's array      
  let promises = keywords.map((keyword, index) =>
    new Promise(resolve => setTimeout(() => {
      for (let page = 1; page <= 5; page++) 
        resolve(request(keyword, page))
      }, index * 100)
   ))

  // Resolve
  await Promise.all(promises).then(() => { 
    // Here is where I hope to be dealing with the fully resolved data to read and write the file 
  })
}

The request function calls the API and then appends the result to a csv file, what I would like to do is when the last promise has been appended to the file, I would like to read that file and manipulated its data, is at this point I am having problems I the csv gets malformed.

I could be down to the way I use fs after should it be sync or async not sure, but would like to know if there is something wrong with this approach on parallel requests.

Any help would be appreciated, many thanks.

Answer 1

You need two Promise.all s - one inside the loop in the new Promise , and one outside waiting for all requests to finish:

const delay = ms =>  new Promise(resolve => setTimeout(resolve, ms));
const pageCount = 5;
const myFunc = async () => {
  const keywords = ['many', 'different', 'type', 'of', 'keywords']
  const promises = keywords.map((keyword, index) =>
    delay(index * 100 * pageCount).then(() => Promise.all(Array.from(
      { length: pageCount },
      (_, i) => delay(100 * (i + 1)).then(() => request(keyword, i + 1))
    )))
  );
  await Promise.all(promises).then(() => { 
    // Here is where I hope to be dealing with the fully resolved data to read and write the file 
  })
}

Since every call needs to be preceeded by another delay, it might be easier to use a for..of loop:

const delay = ms =>  new Promise(resolve => setTimeout(resolve, ms));
const pageCount = 5;
const myFunc = async () => {
  const keywords = ['many', 'different', 'type', 'of', 'keywords']
  for (const keyword of keywords) {
    for (let page = 1; page <= 5; page++) {
      await delay(100);
      await request(keyword, page);
    }
  }
  // Here is where I hope to be dealing with the fully resolved data to read and write the file
};