How to control concurrency in javascript?

Question

I need to control concurrency in a Node.js script I'm making. Currently I'm trying to use npm promise-task-queue but I'm open to other suggestions.

I'm not sure how to implement promise-task-queue into my code. This is my original program:

readURLsfromFile().then( (urls) => {

    urls.reduce( (accumulator, current, i) => {
        return accumulator.then( () => {
            return main(urls[i], i, urls.length)
        })
    }, Promise.resolve())
})

As you can see I'm reading urls from a file, then using .reduce() to run main() in serial on each one of these urls. Serial was too slow though so I need to do it with controlled concurrency.

Here's the code I started to write using promise-task-queue (It's very wrong, I have no idea what I'm doing):

var taskQueue = require("promise-task-queue");

var queue = taskQueue();
var failedRequests = 0;

queue.on("failed:apiRequest", function(task) {
    failedRequests += 1;
});

queue.define("apiRequest", function(task) {
    return Promise.try( () => {
        return main(urls[i], i, urls.length));
    }).then( () => {
        return console.log("DONE!");
    });
}, {
    concurrency: 2
});

Promise.try( () => {
    /* The following queues up the actual task. Note how it returns a Promise! */
    return queue.push("apiRequest", {url: urls[i], iteration: i, amountToDo: urls.length)});
})

As you can see I've put my main() function with its argument after the Promise.try, and I've put my arguments after the return queue.push. Not sure if that's correct or not.

But regardless now I'm stuck, how do I load all the iterations into the queue?

Answer 1

You could use the qew module from npm: https://www.npmjs.com/package/qew .

Install using npm install qew .

To initialise you do

const Qew = require('qew');

const maxConcurrent = 3;
const qew = new Qew(maxConcurrent);

Using the above code qew will now be a queue that allows you to push asynchronous functions onto that will execute with a maximum concurrency of 3.

To push a new async function onto the qew you can do

qew.pushProm(asyncFunc);

So in your case if I understood you correctly you could do something like

readURLsfromFile()
  .then(urls => {
    return Promise.all(urls.map(url => { // wait for all promises to resolve
      return qew.pushProm(() => main(url)); // push function onto queue
    }));
  })
  .then(results => {
    // do stuff with results
  })

In this snippet you are reading urls from a file, and then loading a bunch of functions into the qew one by one and waiting for them all to resolve before doing something with them.

Full disclaimer: I am the author of this package.