在Node.js中将数千个并发的https请求划分为较小的块

Question

I am working on a project where thousands of concurrent https calls are made. My first question is: Where is the concurrence limit? How many calls can be made concurrently? On what does it depends? Secondly, I work on ubuntu system and in most of the cases, the system hangs because of too many requests. How can i divide these thousands of concurrent requests into blocks of something like 50 or 100 requests at a time and on completion of those, the next set of requests are called.

my code looks like:

items.forEach(function(item) {
        request().then(function() {
            if(counter == items.length) {
                deferred.resolve("Success");
                return deferred.promise;
            }
        })
        .catch(function(err) {
            deferred.reject(err);
        });
        counter++;
    });

Items array is mostly more than 10k.

Answer 1

You can use Promise.map concurrency options in bluebird

var Promise = require("bluebird");
var join = Promise.join;
var fs = Promise.promisifyAll(require("fs"));
var concurrency = parseFloat(process.argv[2] || "Infinity");
console.time("reading files");
fs.readdirAsync(".").map(function(fileName) {
    var stat = fs.statAsync(fileName);
    var contents = fs.readFileAsync(fileName).catch(function ignore() {});
    return join(stat, contents, function(stat, contents) {
        return {
            stat: stat,
            fileName: fileName,
            contents: contents
        }
    });
// The return value of .map is a promise that is fulfilled with an array of the mapped values
// That means we only get here after all the files have been statted and their contents read
// into memory. If you need to do more operations per file, they should be chained in the map
// callback for concurrency.
}, {concurrency: concurrency}).call("sort", function(a, b) {
    return a.fileName.localeCompare(b.fileName);
}).then(function() {
    console.timeEnd("reading files");
});

Answer 2

Where is the concurrence limit?

Firstly your all request will go in to the event loop and then one by one in to callstack. So there is no actual limit of this it will start eating your RAM when callstack started filling more and more items.

How many calls can be made concurrently?

You can use series and parallel model -

Where you can to send 50(any number) request in one slot and then another slot and so on.

To make series and parallel request you can use async series and async parallel respectively.

On what does it depends?

Watch the link that I have mentioned in first answer.