限制并发上传

Question

I have a file upload system in my .NET Core site that allows for a user to upload as many files as they want at a time directly to an S3 bucket. A user can upload 1 or many files. The problem that I am having is that when uploading, lets say 1,000 files, the browser does not like to create that many connections and more often than not, files routinely fail to upload. Even with retries enabled, those retries tend to fail since the browser only allows a certain number of concurrent connections. Even worse the browser will lock up.

What I am looking to do is place the files into a queue and only allow 20 files to be actually uploading at any given time (Think of how FileZilla queues items to upload). When a file completes, a new files is added until the queue is exhausted. This way the browser is only creating the connections it needs. I already have it so AutoUpload is set to False and I can place the files into an array to process but the uploadSelectEvent.sender.upload() method enables upload for everything.

Is there a way to pause all the uploads prior to enabling the upload so I can then resume them as needed? Is there a better way to handle this?

Answer 1

I was able to figure this out on my own. Here is how (note this will require some modification to work for your own code and my not be a pure copy and paste answer):

Create an array to hold some data:

/** Files currently uploading */
const uploadQueue: any[] = [];

Capture the FileSuccess event and add this

    // Check the size of the queue
    if (uploadQueue.length < 20) {
        const that = (uploadSuccessEvent.sender as any);
        const module = that._module;
        const upload = module.upload;

        $(".k-file").each((i, x) => {
            //console.log(i, x);
            if (uploadQueue.length < 20) {
                const fileEntry = $(x);
                const started = fileEntry.is(".k-file-progress, .k-file-success, .k-file-error");
                const hasValidationErrors = upload._filesContainValidationErrors(fileEntry.data("fileNames"));

                if (!started && !hasValidationErrors) {
                    uploadQueue.push(fileEntry.data("fileNames")[0].uid);
                    //console.log("fileEntry", fileEntry.data("fileNames")[0].uid);

                    // Start the upload process
                    module.performUpload(fileEntry);
                }
            }
            else { return; }
        });
    }

Create a new function to handle the upload queue:

/**
 * Adds the file to the upload queue and starts the upload.
 * Other files will be loaded via the on success event.
 * @param uploadSelectEvent Select event object.
 */
function queueUpload(uploadSelectEvent: kendo.ui.UploadSelectEvent) {
    //console.log("uploadSelectEvent", uploadSelectEvent);

    // Check the size of the queue
    if (uploadQueue.length < 20) {
        // Start the upload process
        const that = (uploadSelectEvent.sender as any);
        const module = that._module;
        const upload = module.upload;

        //uploadSelectEvent.files.forEach((file, i) => { console.log(i, file); if (uploadQueue.length < 20) { uploadQueue.push(file.uid); } });

        $(".k-file").each((i, x) => {
            //console.log(i, x);
            if (uploadQueue.length < 20) {
                const fileEntry = $(x);
                const started = fileEntry.is(".k-file-progress, .k-file-success, .k-file-error");
                const hasValidationErrors = upload._filesContainValidationErrors(fileEntry.data("fileNames"));

                if (!started && !hasValidationErrors) {
                    uploadQueue.push(fileEntry.data("fileNames")[0].uid);
                    module.performUpload(fileEntry);
                }
            }
            else { return; }
        });
    }
}

Capture the FileSelect event and pass the event to queueUpload .

In the end, you should be limiting the concurrent uploads to 20 at once. It still allows dragging 100's or 1000's of files into the browser (it may still lock up for a sec) but it will only create up to 20 connections at a time. This may not be the most ideal code but it worked for my situation.