Safe way to run multiple async functions in parallel?

Question

I'm writing some code to scan a directory and it occurred to me this may not be the best idea:

files.forEach(async fileName => {
  stat = await lstat(fileName);
});

as I'm going to fire off an lstat for every single file in a directory at the same time. Does anyone know of a "clean" way to do this? I'm thinking a lib that maintains a queue and drains it.

I know some "old" async libraries do this, but I don't know of anything that do it with native async/await calls

Answer 1

Generally, no code does run in parallel so a few hundred open promises shouldn't be a problem.

If you want to run one after another instead a simple for loop will do it :

async function iterate(){
 for(var i=0;i<files.length;i++){
  stat = await lstat(files[i]);
 }
}

To run multiple at once, but not all may do so:

async function iterate(){
 var atonce=10;
 for(var i=0;i<files.length;i+=atonce){
  stats = await Promise.all(files.slice(i,i+atonce).map(file=>lstat(file));
 }
}

Another way would be a few Promise queues:

var current=0;
async function retrieve(){
 if(current>=files.length) return;
 current++;
 await lstat(files[current-1]);
 retrieve();
}
retrieve();//two in parallel
retrieve();

If you want to run all in parallel, may use Promise.all to catch the results ( depends on the usecase):

Promise.all(files.map(async function(file){
 return await lstat(file);
}).then(results=>...);

Answer 2

Are you looking for something like this?

 //sry, but I didn't get up with a beter name function Todo(numParalell = 8){ var todo = [], running = 0; function init(resolve){ if(running < numParalell){ ++running; resolve(); }else{ todo.push(resolve); } } function next(){ if(todo.length){ todo.shift()(); //FIFO //todo.pop()(); //LIFO / FILO }else{ --running; } } return { get numRunning(){ return running }, get numWaiting(){ return todo.length }, append(fn){ if(typeof fn !== "function"){ //fn ain't a function but a value return Promise.resolve(fn); } var promise = new Promise(init).then(fn); promise.then(next, next); return promise; } } } //a queue that runs 4 tasks in paralell var todo = Todo(4); //some task that just kills some time var someAsyncTask = v => new Promise(resolve => setTimeout(resolve, Math.random() * 5000, v)); //add 25 items to the todo-list //and log the start and end time from now var promises = Array(25).fill(Date.now()) .map((t,i) => todo.append(() => { console.log('+starting', i, 'after', Date.now() - t, 'ms'); return someAsyncTask() .then(() => console.log('-finished', i, 'after', Date.now() - t, 'ms')) .then(() => 'resolve to ' + i); })); Promise.all(promises).then(arr => console.log("And the resolved promises:", arr)); 

 .as-console-wrapper{top:0;max-height:100%!important} 

Todo#append() takes a function to perform the async task you want to manage and returns a Promise with the result. Although the function ain't executed immediately, but as soon as there is an empty slot.

Be careful, this code can not resolve dependencies like

var todo = Todo(1);
todo.append(() => todo.append(() => ...)) 

so you might get stuck with the outer task no completing, because it depends on/resolves to the inner one. And the inner one might not be able to start, because there is no free slot on this Todo-list.