Execute at most N tasks in parallel using JS promises or generators

Question

I want to implement a JS class that executes at most N tasks in parallel using JS promises or generators. Something like

class Executor {
    constructor(numberOfMaxTasks) {
    ...
    }

    next(task) {
    ...
    }

    done(onDone) {
    ...
    }
}

....
const executor = new Executor(2);
executor.next(task1).next(task2).next(task3).done(onDone);

task1 and task 2 should be executed in parallel while task3 should wait until one of the previous tasks finises. When all tasks finish onDone callback is executed.

I was trying to implement it using promises but I failed. I'm new to generators and currently have no idea if they can help here. This is for learning purposes mostly, that's why I don't want to use any third party libraries, just native JS. Any hint would be great, thanks in advance!

Answer 1

Recently I had to solve a quiz for an interview similar to what you want to achieve, but based in Node.js .
The key is to take control of amount of tasks being executed at the same time with a class property (in my example, this.running ). So for an array of tasks, you run them through a while loop and check in every loop if you have any available slot (controlled by this.running and LIMIT) and then run the promise.
This code might help you.

    let tasks = [];
    let tasksDone = [];
    const LIMIT = 10;

    class Executor {
        constructor () {
            this.running = 0;
            for (let i = 0; i < 1000; i++) {
                tasks[i] = {
                    id: 'job_' + (i+1),
                    time: ((i % 4) + 1) * 25
                };
            }
        }

        checkConcurrency () {       
            if( this.running > LIMIT ){
                throw new Error('Maximum number of tasks ['+LIMIT+'] reached');
            }
        }

        execute (task) {
            return new Promise((resolve, reject)=>{
                this.running ++;
                this.checkConcurrency();
                setTimeout(()=>{
                    this.checkConcurrency();
                    this.running --;
                    resolve({
                ...task,
                        finished: Date.now()
                    });
                }, task.time);
            })

        }

        run () { 
          this.startTime = Date.now();
          this.executeTasks(tasks.slice(0));
        }

        executeTasks(tasks) {   
          while (this.running < LIMIT && tasks.length > 0) {
            let task = tasks.shift();
            this.execute(task).then( result => {
              tasksDone.push(result);
              if (tasks.length > 0) {
                this.executeTasks(tasks);
              }
            });      
          }   
        }
    }

Answer 2

I think you'd better to use bluebird promise library .

For example, bluebird provides map function which can do what you want:

• in first argument you can specify array of data required for task execution
• in second argument you can specify mapper function which can actually run your task and return the promise
• in third argument you can specify "most N tasks at the time" with { concurrency: N } object.

Note that for bluebird.map function the order of execution doesn't guaranteed. The only guaranteed that result of bluebird.map will be promise fulfilled with array of results in same order.

Using that function you can rewrite your code without Executor class (example is for node.js):

const Promise = require('bluebird')
const os = require('os')
const task1Data = 1000
const task2Data = 5000
const task3Data = 3000

const tasksData = [ task1Data, task2Data, task3Data ]
function taskExecutorMapper(taskData) {
  // here is place for your code that actually runs
  // asynchronous operation based on taskData and returns promise
  // I'll use setTimeout here for emulate such operation
  return new Promise(function(resolve) {
    setTimeout(resolve, taskData)
  }
}
const tasksPromise = Promise.map(
  tasksData,
  taskExecutionMapper,
  { concurrency: os.cpus().length })
  .then(onDone)

Hope this helps!