Delay after uploading large files with Jquery BlueImp uploader

Question

I'm able to successfully upload large files (tested up to 8GB) via the BlueImp Jquery Uploader plugin.

My PHP settings are:

upload_max_filesize 8192M
post_max_size 8192M
max_execution_time 200
max_input_time 200
memory_limit 8192M

The problem I'm having is that when large files (2gb or larger) finish uploading, the progress bar hangs on 100% and takes considerable time to "finish". I'm not sure what post processing is happening, other than piecing the file chunks together.

I've tried adjusting the chunk size in the UploadHandler.php file, and it seems to improve things slightly when increased (eg from 10mb to 500mb chunks) but the delay is still there. There doesn't seem to be any improvement when disabling chunked uploads altogether (0), and I'm not sure of the potential ramifications of doing this also.

On a 2gb file, the delay is around 20 seconds, but on a 4gb file, it's around 2 minutes. A 7gb file takes around 3-4 minutes, and sometimes times out. This leaves the user waiting, unaware of what is happening as the progress bar has finished at 100% by this point.

Does anyone have any insight as to what this might be, or how to go about troubleshooting it? I suspected that the file in /tmp might be being copied rather than moved, but there's no sign of this in the php file as far as I can tell.

Boosting the CPU and RAM on my server VM improves things a little, though running the "top" command during this process reveals that CPU and RAM don't appear to be extensively exhausted (0-2%, compared to 90-100% during actual upload).

Many thanks in advance.

Answer 1

I don't know very much about it , i search on google and i found some , i think it might help you.

Normally “onError” event is fired when exception occurs on Server or Client. In the mentioned situation there is no exception thrown until the Server itself does not time out.

One possibility is to monitor the upload status and set JavaScript timeout or counter in uploading event that will cancel the upload after reached time limit.

All upload status and error codes are shown here - http://help.infragistics.com/NetAdvantage/jQuery/2013.1/CLR4.0?page=igUpload_Using_Client_Side_Events.html

You can monitor these with “fileStatus” argument in uploading event.

Link Which will help you

1. Link1
2. Link2
3. Link3

Answer 2

What this might be :

jQuery FileUpload registers the request progress event. This is the event when the browser is sending data to the server, not the confirmation event from the server. There is a delay between the browser sending and the server confirming reception. See Can onprogress functionality be added to jQuery.ajax() by using xhrFields? for the two events. However, I could not resolve this for me and used a work-around, so I cannot fully confirm this assumption.

How to go about troubleshooting it :

Use your Browser's debugging tools and set a breakpoint around here in the code to get an understanding of variables available: https://github.com/blueimp/jQuery-File-Upload/blob/master/js/jquery.fileupload.js#L372

Once you are sure what you want to see, best is likely to print information with console.log and check your Browser's console output.

Work-arounds :

If you are only interested to see whether all downloads have finished and display to the user that something is still going on, you may want to check out stop / done events, or also fileupload('active'), see here: Finish of Multiple file upload

Answer 3

After much troubleshooting and debugging of this issue, I've found what I believe to be the cause, and a better workaround/solution. It's a bit “hacky” (I'm an amateur dev!), so I'm open to any suggestions to improve this, though it does seem to work for me.

The files uploaded from the client are actually uploaded to this temp location on the server: /tmp/systemd-private- random-number -httpd.service- random-number /tmp/sess_ php-session-ID

Once the uploads have completed on the client side, the UI progress bar reaches 100% and remains so whilst the server is processing the files.

The processing on the server side involves moving (copying, then deleting) the file from the above /tmp location to the relevant blueimp location. (Assuming “user_dirs” is enabled/true in the blueimp options, the default destination is: /var/www/html/server/php/files/ php-session-id / ).

This copying process can take a significant amount of time, particularly on files larger than 2gb or thereabouts. Once the server processing has completed, blueimp triggers the “fileuploaddone” callback, and the UI updates to a completed state.

My aim was to provide some interactive UI feedback at this point (rather than hanging at 90% like the other workaround). My environment is capable of very large uploads (10gb+), so I didn't find it acceptable not to provide any user feedback for what could be several minutes of file processing (with the user thinking the site has crashed, and closing the browser etc.).

My Workaround:

The first problem I encountered was that there doesn't appear to be a blueimp callback for the point when the file uploads have completed on the client side, and the server processing begins. So I worked around this by creating a function (in main.js) to display my custom “processing div” after data.loaded matches data.total:

$('#fileupload').bind('fileuploadprogressall', function (e, data) { console.log(data); 
if (data.loaded == data.total) { 
setTimeout(function(){
$("#processwarn").fadeTo(300, 1);
},3000);
}})

My custom div that sits below the progress bar info is another php page that's refreshed (with ajax.load) every couple of seconds and calculates the total file size of all the files in the current session upload folder (the directory total size didn't seem to provide accurate results):

// Calculate size of files being processed
$filePath = "/var/www/html/server/php/files/*php-session-id*/";

$total = 0;
$d = new RecursiveIteratorIterator(
  new RecursiveDirectoryIterator($filePath), 
  RecursiveIteratorIterator::SELF_FIRST
);

foreach($d as $file){
  $total += $file->getSize();
}

// Convert to readable format
if ($total >= 1073741824)
{
    $total = number_format($total / 1073741824, 2) . ' GB';
}
elseif ($total >= 1048576)
{
    $total = number_format($total / 1048576, 2) . ' MB';
}
elseif ($total >= 1024)
{
    $total = number_format($total / 1024, 2) . ' KB';
}
elseif ($total > 1)
{
    $total = $total . ' bytes';
}
elseif ($total == 1)
{
    $total = $total . ' byte';
}
else
{
    $total = '0 bytes';
}

// Display spinner gif and size of file currently being processed
echo "<img src=\"img/spinner.gif\" height=\"20px\" width=\"20px\">&nbsp;&nbsp;Please wait, processing files: &nbsp;&nbsp;$total";

The result looks something like this:

Edit: (after some more work) - With bootstrap progress bar added instead:

Additional point noticed from testing:

With chunking enabled (for example set to 1gb (maxChunkSize: 1000000000) ), the problem almost disappears, as the processing time appears drastically reduced to the user, as the "copying" processing occurs at the point of chunking (every 1gb in this example). When the final processing then occurs, the server only has to "rechunk"/copy the last remaining 1gb over.
I also experienced quicker overall upload times due to this. In hindsight, this may well be an easier/more effective solution for many.