Limit the number of concurrent child processes spawned in a loop in Node.js

Question

I am trying to use child_process.spawn to call a CLI tool in a for loop with different arguments each time it's called. So far so good, but if I want to introduce a maximum number of the child processes and only continue spawning new processes when a previous process closes, I run into troubles. I wanted to put the for loop in halt with an infinite while loop when the limited child processes amount is achieved. However, the child processes seem to never fire a 'close' event.

Using ls as an example (sorry I can't think of a good, long lasting command that auto quits after a while):

const { spawn } = require("child_process");

const max = 3;
let current = 0;

// dirsToVisit is an array of paths
for (let i = 0; i < dirsToVisit.length; i++) {
  // if already running 3 ls, wait till one closes
  while (current >= max) {}
  current++;
  lsCommand(dirsToVisit[i]);
}

function lsCommand(dir) {
  const ls = spawn("ls", [dir]);
  ls.on("close", code => {
    current--;
    console.log(`Finished with code ${code}`);
  });
}

This code above never exits, the string to be logged in the console when the child process exits never printed on screen. If I remove the while loop, all child processes finish in the end without issues, but there will be no limit how many processes are allowed at the same time.

Why is my code not working, and how do I correctly limit the number of child process spawned in a loop? Any help would be appreciated!

Answer 1

Your code doesn't work because lsCommand() is non-blocking, asynchronous. All it does is initiate the spawn operations and then immediately return. So, your for loop starts to run, then your while loops run in the first iteration of the for loop and starts max lsCommand() calls and then it quits. Subsequent iterations of the for loop have nothing else to do because max lsCommand() calls are already running. So, since lsCommand() is non-blocking, your for loop finishes and all it did was start max lsCommand() operations and then your loop is done. What you have to do is you have to watch for the completion of each lsCommand() by monitoring ls.on('close')` and then when each one finishes, you can then start another one. You can see how I do that in my code below.

You can do something like this where you create an internal function that has a loop to start up processes up to your limit and then you just keep calling that function each time a spawn operation finishes (which will fire up one more each time one finishes):

function listDirs(dirsToVisit, maxAtOnce) {
    let numRunning = 0;
    let index = 0;

    function runMore() {
        // while we need to start more, start more of them
        while (numRunning < maxAtOnce && index < dirsToVisit.length) {
            ++numRunning;
            const ls = spawn("ls", [dirsToVisit[index++]]);
            ls.on("close", code => {
                --numRunning;
                console.log(`Finished with code ${code}`);
                runMore();
            }).on("error", err => {
                --numRunning;
                runMore();
            });
        }
        if (numRunning === 0) {
            // all done with all requests here
        }
    }
    runMore();
}

For some more generic implementations, see these:

Loop through an api get request with variable URL

Promise.all consumes all my RAM

Javascript - how to control how many promises access network in parallel

Nodejs: Async request with a list of URL