发送数千个提取请求会使浏览器崩溃。 记不清

Question

I was tasked with transferring a large portion of data using javascript and an API from one database to another. Yes I understand that there are better ways of accomplishing this task, but I was asked to try this method.

I wrote some javascript that makes a GET call to an api that returns an array of data, which I then turnaround and make calls to another api to send this data as individual POST requests.

What I have written so far seems to works fairly well, and I have been able to send over 50k individual POST requests without any errors. But I am having trouble when the number of POST requests increases past around 100k. I end up running out of memory and the browser crashes.

From what I understand so far about promises, is that there may be an issue where promises (or something else?) are still kept in heap memory after they are resolved, which results in running out of memory after too many requests.

I've tried 3 different methods to get all the records to POST successfully after searching for the past couple days. This has included using Bluebirds Promise.map, as well as breaking up the array into chunks first before sending them as POST requests. Each method seems to work up until it has processed about 100k records before it crashes.

async function amGetRequest(controllerName) {

    try{
    const amURL = "http://localhost:8081/api/" + controllerName;

    const amResponse = await fetch(amURL, {
        "method": "GET",
    });

    return await amResponse.json();

    } catch (err) {
        closeModal()
        console.error(err)
    }
};


async function brmPostRequest(controllerName, body) {

    const brmURL = urlBuilderBRM(controllerName);
    const headers = headerBuilderBRM();

    try {
        await fetch(brmURL, {
            "method": "POST",
            "headers": headers,
            "body": JSON.stringify(body)
        });
    }
    catch(error) {
        closeModal()
        console.error(error);
    };
};


//V1.0 Send one by one and resolve all promises at the end.

const amResult = await amGetRequest(controllerName); //(returns an array of ~245,000 records)

let promiseArray = [];

for (let i = 0; i < amResult.length; i++) {
    promiseArray.push(await brmPostRequest(controllerName, amResult[i]));
};

const postResults = await Promise.all(promiseArray);



//V2.0 Use bluebirds Promise.map with concurrency set to 100

const amResult = await amGetRequest(controllerName); //(returns an array of ~245,000 records)

const postResults = Promise.map(amResult, async data => {
    await brmPostRequest(controllerName, data);
    return Promise.resolve();
}, {concurrency: 100});



//V3.0 Chunk array into max 1000 records and resolve 1000 promises before looping to the next 1000 records

const amResult = await amGetRequest(controllerName); //(returns an array of ~245,000 records)

const numPasses = Math.ceil(amResult.length / 1000);

for (let i=0; i <= numPasses; i++) {
    let subset = amResult.splice(0,1000);
    let promises = subset.map(async (record) => {
        await brmPostRequest(controllerName, record);
    });

    await Promise.all(promises);

    subset.length = 0; //clear out temp array before looping again
};

Is there something that I am missing about getting these promises cleared out of memory after they have been resolved?

Or perhaps a better method of accomplishing this task?

Edit: Disclaimer - I'm still fairly new to JS and still learning.

Answer 1

"Well-lll ... you're gonna need to put a throttle on this thing!"

Without (pardon me ...) attempting to dive too deeply into your code, "no matter how many records you need to transfer, you need to control the number of requests that the browser attempts to do at any one time."

What's probably happening right now is that you're stacking up hundreds or thousands of "promised" requests in local memory – but, how many requests can the browser actually transmit at once? That should govern the number of requests that the browser actually attempts to do. As each reply is returned, your software then decides whether to start another request and if so for which record.

Conceptually, you have so-many "worker bees," according to the number of actual network requests your browser can simultaneously do. Your software never attempts to launch more simultaneous requests than that: it simply launches one new request as each one request is completed. Each request, upon completion, triggers code that decides to launch the next one.

So – you never are "sending thousands of fetch requests." You're probably sending only a handful at a time, even though, in this you-controlled manner, "thousands of requests do eventually get sent."

Answer 2

As you are not intereted in the values delivered by brmPostRequest() , there's no point mapping the original array; neither the promises nor the results need to be acumulated.

Not doing so will save memory and may allow progress beyond the 100k sticking point.

async function foo() {
    const amResult = await amGetRequest(controllerName);
    let counts = { 'successes': 0, 'errors': 0 };
    for (let i = 0; i < amResult.length; i++) {
        try {
            await brmPostRequest(controllerName, amResult[i]);
            counts.successes += 1;
        } catch(err) {
            counts.errors += 1;
        }
    };
    const console.log(counts);
}