Promise All in Node.js with a forEach loop

Question

I have a function that reads a directory and copies and creates a new file within that directory.

function createFiles (countryCode) {
  fs.readdir('./app/data', (err, directories) => {
    if (err) {
      console.log(err)
    } else {
      directories.forEach((directory) => {
        fs.readdir(`./app/data/${directory}`, (err, files) => {
          if (err) console.log(err)
          console.log(`Creating ${countryCode}.yml for ${directory}`)
          fs.createReadStream(`./app/data/${directory}/en.yml`).pipe(fs.createWriteStream(`./app/data/${directory}/${countryCode}.yml`))
        })
      })
    }
  })
}

How do I do this using promises or Promise All to resolve when it's complete?

Answer 1

First, you need to wrap each file stream in a promise that resolves when the stream emits the finish event:

new Promise((resolve, reject) => {
  fs.createReadStream(`./app/data/${directory}/en.yml`).pipe(
    fs.createWriteStream(`./app/data/${directory}/${countryCode}.yml`)
  ).on('finish', resolve);
});

The you need to collect these promises in an array. This is done by using map() instead of forEach() and returning the promise:

var promises = directories.map((directory) => {
  ...
  return new Promise((resolve, reject) => {
    fs.createReadStream( ...
    ...
  });
});

Now you have a collection of promises that you can wrap with Promise.all() and use with a handler when all the wrapped promises have resolved:

Promise.all(promises).then(completeFunction);

Answer 2

In recent versions of Node (8.0.0 and later), there's a new util.promisify function you can use to get a promise. Here's how we might use it:

// Of course we'll need to require important modules before doing anything
// else.
const util = require('util')
const fs = require('fs')

// We use the "promisify" function to make calling promisifiedReaddir
// return a promise.
const promisifiedReaddir = util.promisify(fs.readdir)

// (You don't need to name the variable util.promisify promisifiedXYZ -
// you could just do `const readdir = util.promisify(fs.readdir)` - but
// I call it promisifiedReaddir here for clarity.

function createFiles(countryCode) {
  // Since we're using our promisified readdir function, we'll be storing
  // a Promise inside of the readdirPromise variable..
  const readdirPromise = promisifiedReaddir('./app/data')

  // ..then we can make something happen when the promise finishes (i.e.
  // when we get the list of directories) by using .then():
  return readdirPromise.then(directories => {
    // (Note that we only get the parameter `directories` here, with no `err`.
    // That's because promises have their own way of dealing with errors;
    // try looking up on "promise rejection" and "promise error catching".)

    // We can't use a forEach loop here, because forEach doesn't know how to
    // deal with promises. Instead we'll use a Promise.all with an array of
    // promises.

    // Using the .map() method is a great way to turn our list of directories
    // into a list of promises; read up on "array map" if you aren't sure how
    // it works.
    const promises = directory.map(directory => {
      // Since we want an array of promises, we'll need to `return` a promise
      // here. We'll use our promisifiedReaddir function for that; it already
      // returns a promise, conveniently.
      return promisifiedReaddir(`./app/data/${directory}`).then(files => {
        // (For now, let's pretend we have a "copy file" function that returns
        // a promise. We'll actually make that function later!)
        return copyFile(`./app/data/${directory}/en.yml`, `./app/data/${directory}/${countryCode}.yml`)
      })
    })

    // Now that we've got our array of promises, we actually need to turn them
    // into ONE promise, that completes when all of its "children" promises
    // are completed. Luckily there's a function in JavaScript that's made to
    // do just that - Promise.all:
    const allPromise = Promies.all(promises)

    // Now if we do a .then() on allPromise, the function we passed to .then()
    // would only be called when ALL promises are finished. (The function
    // would get an array of all the values in `promises` in order, but since
    // we're just copying files, those values are irrelevant. And again, don't
    // worry about errors!)

    // Since we've made our allPromise which does what we want, we can return
    // it, and we're done:
    return allPromise
  })
}

Okay, but, there's probably still a few things that might be puzzling you..

What about errors? I kept saying that you don't need to worry about them, but it is good to know a little about them. Basically, in promise-terms, when an error happens inside of a util.promisify 'd function, we say that that promise rejects . Rejected promises behave mostly the same way you'd expect errors to; they throw an error message and stop whatever promise they're in. So if one of our promisifiedReaddir calls rejects, it'll stop the whole createFiles function.

What about that copyFile function? Well, we have two options:

1. Use somebody else's function. No need to re-invent the wheel! quickly-copy-file looks to be a good module (plus, it returns a promise, which is useful for us).

2. Program it ourselves.

Programming it ourselves isn't too hard, actually, but it takes a little bit more than simply using util.promisify :

function copyFile(from, to) {
  // Hmm.. we want to copy a file. We already know how to do that in normal
  // JavaScript - we'd just use a createReadStream and pipe that into a
  // createWriteStream. But we need to return a promise for our code to work
  // like we want it to.

  // This means we'll have to make our own hand-made promise. Thankfully,
  // that's not actually too difficult..

  return new Promise((resolve, reject) => {
    // Yikes! What's THIS code mean?
    // Well, it literally says we're returning a new Promise object, with a
    // function given to it as an argument. This function takes two arguments
    // of its own: "resolve" and "reject". We'll look at them separately
    // (but maybe you can guess what they mean already!).

    // We do still need to create our read and write streams like we always do
    // when copying files:
    const readStream = fs.createReadStream(from)
    const writeStream = fs.createWriteStream(to)

    // And we need to pipe the read stream into the write stream (again, as
    // usual):
    readStream.pipe(writeStream)

    // ..But now we need to figure out how to tell the promise when we're done
    // copying the files.

    // Well, we'll start by doing *something* when the pipe operation is
    // finished. That's simple enough; we'll just set up an event listener:
    writeStream.on('close', () => {
      // Remember the "resolve" and "reject" functions we got earlier? Well, we
      // can use them to tell the promise when we're done. So we'll do that here:
      resolve()
    })

    // Okay, but what about errors? What if, for some reason, the pipe fails?
    // That's simple enough to deal with too, if you know how. Remember how we
    // learned a little on rejected promises, earlier? Since we're making
    // our own Promise object, we'll need to create that rejection ourself
    // (if anything goes wrong).

    writeStream.on('error', err => {
      // We'll use the "reject" argument we were given to show that something
      // inside the promise failed. We can specify what that something is by
      // passing the error object (which we get passed to our event listener,
      // as usual).
      reject(err)
    })

    // ..And we'll do the same in case our read stream fails, just in case:
    readStream.on('error', err => {
      reject(err)
    })

    // And now we're done! We've created our own hand-made promise-returning
    // function, which we can use in our `createFiles` function that we wrote
    // earlier.
  })
}

..And here's all the finished code, so that you can review it yourself:

const util = require('util')
const fs = require('fs')

const promisifiedReaddir = util.promisify(fs.readdir)

function createFiles(countryCode) {
  const readdirPromise = promisifiedReaddir('./app/data')

  return readdirPromise.then(directories => {
    const promises = directory.map(directory => {
      return promisifiedReaddir(`./app/data/${directory}`).then(files => {
        return copyFile(`./app/data/${directory}/en.yml`, `./app/data/${directory}/${countryCode}.yml`)
      })
    })

    const allPromise = Promies.all(promises)

    return allPromise
  })
}

function copyFile(from, to) {
  return new Promise((resolve, reject) => {
    const readStream = fs.createReadStream(from)
    const writeStream = fs.createWriteStream(to)
    readStream.pipe(writeStream)

    writeStream.on('close', () => {
      resolve()
    })

    writeStream.on('error', err => {
      reject(err)
    })

    readStream.on('error', err => {
      reject(err)
    })
  })
}

Of course, this implementation isn't perfect . You could improve it by looking at other implementations - for example this one destroys the read and write streams when an error occurs, which is a bit cleaner than our method (which doesn't do that). The most reliable way would probably to go with the module I linked earlier!

---

I highly recommend you watch funfunfunction's video on promises . It explains how promises work in general, how to use Promise.all , and more; and he's almost certainly better at explaining this whole concept than I am!

Answer 3

First, create a function that returns a promise:

function processDirectory(directory) {
  return new Promise((resolve, reject) => {
    fs.readdir(`./app/data/${directory}`, (err, files) => {
      if (err) reject(err);

      console.log(`Creating ${countryCode}.yml for ${directory}`);

      fs.createReadStream(`./app/data/${directory}/en.yml`)
        .pipe(fs.createWriteStream(`./app/data/${directory}/${countryCode}.yml`))
        .on('finish', resolve);
    });
  });
}

Then use Promise.all:

Promise.all(directories.map(processDirectory))
  .then(...)
  .catch(...);