Uploading chunked files directly to Amazon s3

Question

I'm looking for any straight forward examples on uploading directly to Amazon s3 in chunks without any server side processing (aside signing the request)

I've looked into many options and so far all examples either address just the chunking from the server or just sending to s3 from browser as a single put, or are so old they just don't work anymore.

Highest hope was Plupload, but I can't find any documentation for splitting large files into chunks, at least not in the amazon example.

Ultimately the goal is to send a 500mb file to s3 in under 5 seconds. Using the server in php I am able to get close to 10 seconds, but the client wants to avoid server processing entirely and go straight to s3, so those are the rules I'm bound by.

I find Amazon documentation highly convoluted and difficult to follow. Does anyone out there know of a complete example of browser chunking to s3?

Basic idea of Plupload settings:

$("#uploader").plupload({
    runtimes : 'html5,flash,silverlight',
    url : 'http://<?php echo $bucket; ?>.s3.amazonaws.com/',

    multipart: true,
    multipart_params: {
        'key': '${filename}', // use filename as a key
        'Filename': '${filename}', // adding this to keep consistency across the runtimes
        'acl': 'public-read',
        'Content-Type': 'Binary/Octet-Stream',
        'AWSAccessKeyId' : '<?php echo $accessKeyId; ?>',       
        'policy': '<?php echo $policy; ?>',
        'signature': '<?php echo $signature; ?>',
        'Multiple_queues': True,
    },
    file_data_name: 'file',
    filters : {
        // Maximum file size (apparently not per chunk)
        max_file_size : '5mb',
    },
    flash_swf_url : '../js/Moxie.swf',
    silverlight_xap_url : '../js/Moxie.xap'
});

It seems multipart is not multiple chunks, but maybe multiple files total. Not sure, I only need to send one.

Any input is appreciated.

Answer 1

Plupload does support chunked uploads so all you need to do is configure it properly:

var uploader = new plupload.Uploader({
    browse_button: 'browse', // this can be an id of a DOM element or the DOM element itself
    url: 'upload.php',
    chunk_size: '200kb',
    max_retries: 3
});

The last two lines of the above config will set chunking.