Recursive Promise-based directory reading

Question

I have a library that scans a directory for files on a remote server. It returns a Promise like this:

client.scanRemoteDirectory(path)
  .then(files => { 

    console.log(files)

  })

I'm trying to write a recursive method to scan directories and subdirectories too. But I'm running into some async issues. My function is like this:

const scanDir(path) {

  // Scan the remote directory for files and sub-directories
  return client.scanRemoteDirectory(path)
    .then(files => {

      for (const file of files) {
        // If a sub-directory is found, scan it too
        if (file.type === 'directory') {

          return scanDir(file.path) // Recursive call

        }
      }
    })
}

const scanDir('some/path')
  .then(() => {
    console.log('done')
  })

This works however because of the return in front of the scanDir() recursive method call, this results in the method only scanning the first subdirectory in each directory and skipping the rest.

So for example if the structure is something like this:

/some/path
/some/path/dirA
/some/path/dirA/subdirA
/some/path/dirB
/some/path/dirB/subdirB

The above method will only scan:

/some/path
/some/path/dirA
/some/path/subdirA

It will skip dirB and it's children altogether since the method finds dirA first.

If I simply remove the return from the return scanDir(...) call, then it scans everything just fine. But then my final console.log('done') happens too soon because it's async.

So how do I solve this problem? What is the proper recursive Promise approach where I can still preserve async but also scan every subdirectory recursively?

Answer 1

You might want to use Promise.all in this situation to run your 'sub' promises in parallel, for example:

function scanDir(path) {

    return client.scanRemoteDirectory(path)
        .then(all => {
            const files = all.where(file => file.type !== 'directory);
            const dirs = all.where(file => file.type === 'directory);
            return Promise.all(dirs.map(dir => scanDir(dir.path)) // Execute all 'sub' promises in parallel.
                .then(subFiles => {
                    return files.concat(subFiles);
                });
        });
}

Alternatively you could use the reduce function to run your 'sub' promises in sequence:

function scanDir(path) {

    return client.scanRemoteDirectory(path)
        .then(all => {
            const files = all.where(file => file.type !== 'directory);
            const dirs = all.where(file => file.type === 'directory);
            return dirs.reduce((prevPromise, dir) => { // Execute all 'sub' promises in sequence.
                    return prevPromise.then(output => {
                        return scanDir(dir.path)
                            .then(files => {
                                return output.concat(files);
                            });
                    });
                }, Promise.resolve(files));
        });
}

Async / await is definitely the easiest solution to read:

async function scanDir(path) {

    const output = [];
    const files = await client.scanRemoteDirectory(path);
    for (const file of files) {
        if (file.type !== 'directory') {
            output.push(file);
            continue;
        }

        const subFiles = await scanDir(file.path);
        output = output.concat(subFiles);       
    }

    return output;
}

Answer 2

I would make the then handler async so you can use await in the loop:

 const scanDir(path) {
  // Scan the remote directory for files and sub-directories
  return client.scanRemoteDirectory(path)
    .then(async files => {
       for (const file of files) {
          // If a sub-directory is found, scan it too
          if (file.type === 'directory') {
            await scanDir(file.path) // Recursive call
          }
      }
   })
 }