Javascript - Chaining 2 (or more) arrays of promises

Question

I have some code that does this: First scrape this array of webpages. After that, scrape another array of webpages.

The following code does what I expect:

let bays=[];
let promises=promisesN=[];

for (let y=2019;y>=2015;y--) 
    promises.push(new Promise(resolve=>
        curl.get(`/*url*/${y}.html`,null, (error,resp,body)=> 
            resp.statusCode==200? resolve(parse(body)):reject(error)
    )));
Promise.all(promises).then(()=>{
    bays.forEach(bay=>{
        if (bay.no.match(/\d+/)<=103) return;
        promisesN.push(new Promise(resolve=>
            curl.get(`/*url*/${bay.code}/`,null, (error,resp,body)=> 
                resp.statusCode==200? resolve(image(bey,body)):reject(error)
    )))});
    Promise.all(promisesN).then(()=>{
        bays.sort((a,b)=>{return parseInt(a.no.match(/\d+/))<parseInt(b.no.match(/\d+/))? -1:1});
        console.log(bays);
    });
}).catch(error=>console.log(error));`

So I've read you can write a simplier nesting-free syntax:

doSomething()
.then(function(result) {
  return doSomethingElse(result);
})
.then(function(newResult) {
  return doThirdThing(newResult);
})
.then(function(finalResult) {
  console.log('Got the final result: ' + finalResult);
})
.catch(failureCallback);

How to apply this to the code above?

Answer 1

correctness

let promises=promisesN=[];

This is really incorrect. It makes both variables reference the same array, and makes promisesN an implicit global. The fact that it appears to work means you aren't in strict mode. Always use strict mode . The correct version of what you intended is:

let promises = [];
let promisesN = [];

cleanliness

new Promise(resolve=>
    curl.get(`/*url*/${y}.html`,null, (error,resp,body)=> 
        resp.statusCode==200? resolve(parse(body)):reject(error)
))

You're repeating this pattern, so make it into a function, or use a package that does the job for you, like request-promise[-native] or axios . (Also, please show your real code. reject isn't defined here.)

const getAsync = url => new Promise((resolve, reject) => {
    curl.get(url, null, (error, resp, body) => {
        if (resp.statusCode === 200) {
            resolve(body);
        } else {
            reject(error);
        }
    });
});

Notice how you're free to make the function more readable when it isn't repeated, and to extend it later.

let promises = [];
let promisesN = [];

for (let y = 2019; y >= 2015; y--) {
    promises.push(getAsync(`/*url*/${y}.html`).then(parse));
}

Promise.all(promises).then(bays => {
    bays.forEach(bay => {
        if (bay.no.match(/\d+/) <= 103) return;
        promisesN.push(getAsync(`/*url*/${bay.code}/`).then(body => image(bay, body)));
    });

    Promise.all(promisesN).then(() => {
        bays.sort((a, b) => {return parseInt(a.no.match(/\d+/)) < parseInt(b.no.match(/\d+/)) ? -1 : 1;});
        console.log(bays);
    });
}).catch(error => console.log(error));

I had to take a few guesses at what your real code looks like again, because you're surely doing something with the resolved value of Promise.all(promises) . It doesn't have any easily-accessible side-effects. bey also seemed likely enough to be bay .

Now you can give promisesN a more appropriate scope:

let promises = [];

for (let y = 2019; y >= 2015; y--) {
    promises.push(getAsync(`/*url*/${y}.html`).then(parse));
}

Promise.all(promises).then(bays => {
    let promisesN = bays
        .filter(bay => bay.no.match(/\d+/) > 103)
        .map(bay => getAsync(`/*url*/${bay.code}/`).then(body => image(bay, body)));

    Promise.all(promisesN).then(() => {
        bays.sort((a, b) => {return parseInt(a.no.match(/\d+/)) < parseInt(b.no.match(/\d+/)) ? -1 : 1;});
        console.log(bays);
    });
}).catch(error => console.log(error));

and use an expression-bodied arrow function where appropriate, since you're already using them whenever they aren't appropriate:

bays.sort((a, b) => parseInt(a.no.match(/\d+/)) < parseInt(b.no.match(/\d+/)) ? -1 : 1);

Now, if my guess about bays is right, then you can't unnest. If it comes from somewhere else then you can. Normally I would leave a comment about that but I already wrote all this, so… please clarify that for further cleanup.

Answer 2

If you're looking to simplify your code, you might consider the use of async/await instead of promises.

The async/await syntax will greatly simplify the presentation and ease comprehension of the code, especially given that your logic relies on asynchronous iteration of arrays.

Consider the following code revision of your code:

/* Define local helper that wraps curl() in async function declaration */
function async doRequest(url) {

    return (await new Promise(resolve=> curl.get(url, null, (error,resp,body) => 
        resp.statusCode==200 ? resolve(res) : reject(error))))
}

/* Re-define simplified scrape logic using await/async */
function async doScrape() {

    try {

        var bays = []

        /* Iterate date range asynchronously */
        for (let y=2019; y>=2015; y--)  {

            /* Use doRequest helper function to fetch html */
            const response = await doRequest(`/*url*/${y}.html`)

            const bay = parse(response)
            bays.push(bay)
        }

        /* Iterate bays array that was obtained */
        for(const bay of bays) {

            /* Use doRequest helper again to fetch data */
            const response = await doRequest(`/*url*/${bay.code}/`)

            /* Await may not be needed here */
            await image(bay, response) 
        }

        /* Perform your sort (which is non asynchronous) */
        bays.sort((a,b)=> parseInt(a.no.match(/\d+/))<parseInt(b.no.match(/\d+/))? -1:1);

        console.log("Result", bays);
    }
    catch(err) {

        /* If something goes wrong we arrive here - this is 
        essentially equivalent to your catch() block */
        console.error('Scrape failed', err);
    }
}

/* Usage */
doScrape()

Hope that helps!

Answer 3

Not entirely sure if this is what you want, but I've separated your code out a bit because I found it easier for me to read.

let bays = [];
let promises = [];
let promisesN = [];

for (let y = 2019; y >= 2015; y--) {
  const promiseOne = new Promise((resolve, reject) => {
    return curl.get(`/*url*/${y}.html`, null, (error, resp, body) => {
      resp.statusCode === 200 ? resolve(parse(body)) : reject(error);
    });
  });
  promises.push(promiseOne);
}

Promise.all(promises)
  .then(() => {
    bays.forEach((bay) => {
      if (bay.no.match(/\d+/) <= 103) {
        return;
      }

      const promiseTwo = new Promise((resolve, reject) => {
        return curl.get(`/*url*/${bay.code}/`, null, (error, resp, body) => {
          resp.statusCode === 200 ? resolve(image(bay, body)) : reject(error);
        });
      });

      promisesN.push(promiseTwo);
    });

    return Promise.all(promisesN);
  })
  .then(() => {
    bays.sort((a, b) => {
      return parseInt(a.no.match(/\d+/), 10) < parseInt(b.no.match(/\d+/), 10) ? -1 : 1;
    });
    console.log(bays);
  })
  .catch((error) => {
    console.log(error);
  });

I am wondering though, you are firing the promises instantly on each iteration of your for loop. This might be intentional, but it means if those promises resolve before the code gets to execute Promise.all you may run into issues. I personally would do something like, eg const promiseOne = () => somePromise, that way you can create a bunch of promises, and then once they're all created, map over that array and fire them at once. Same thing goes for the second promises.

Not sure if this is helpful, let me know if it is. Feel free to ask more questions too.