快速发出 10,000 个 HTTP 请求

Question

I would like to make 10,000 concurrent HTTP requests. I am currently doing it by using Promise.all . However, I seem to be rate limited in some way, it takes around 15-30 mins to complete all 10,000 requests. Is there something in axios or in the http requests in node that is limiting me? How can I raise the limt if there is one?

const axios = require('axios');

function http_request(url) {
    return new Promise(async (resolve) => {
        await axios.get(url);
        // -- DO STUFF
        resolve();
    });
}

async function many_requests(num_requests) {
    let all_promises = [];
    for (let i = 0; i < num_requests; i++) {
        let url = 'https://someurl.com/' + i;
        let promise = http_request(url);
        all_promises.push(promise);
    }
    return Promise.all(all_promises);
}

async function run() {
    await many_requests(10000);
}

run();

Answer 1

In Node.js there are two types of threads: one Event Loop (aka the main loop, main thread, event thread, etc.), and a pool of k Workers in a Worker Pool (aka the threadpool).

...

The Worker Pool of Node.js is implemented in libuv (docs), which exposes a general task submission API.

source

libuv

Default UV_THREADPOOLSIZE is 4. You can set UV_THREADPOOLSIZE as link. Limit of it depend on os, you need check your os:

set UV_THREADPOOL_SIZE