Direct upload to amazon S3 using jquery.fileupload

Question

In my app I would like to be able to upload files directly from the browser to my AWS S3 bucket. My backend is rails but I'd like to avoid the extra hop to my server, and avoid using gems like paperclip, carrierwave, carrierwave_direct etc, to keep it simple. I'm roughly following this tutorial from heroku.

I am using aws-sdk gem and the jquery.fileupload.js lib.

The problem is that when I try to do the upload, I get back a 400 bad request back from AWS.

I don't think that it is a CORS issue. I have CORS configured on my bucket and I can see a successful OPTIONS request followed by the POST request for the file upload, which returns the 400 bad request.

The following is a simplified demo that replicates the problem.

This is the controller action. It generates an AWS::S3::PresignedPost object so the view can use it post files directly to S3.

  def new
    Aws.config.update({
      region: 'us-east-1',
      credentials: Aws::Credentials.new('[FILTERED]', '[FILTERED]'),
    })
    s3 = Aws::S3::Resource.new
    bucket = s3.bucket('mybucket')
    @presigned_post = bucket.presigned_post(key: "attachments/#{SecureRandom.uuid}/${filename}")
    @thing = Thing.new
  end

This is the view, new.html.erb , which is rendered by the the above, with the upload form and the javascript to handle the upload.

<div class='container'>
  <%= form_for(@thing, html: { class: 'direct_upload' }) do |f| %>
    <%= f.label 'Thing' %>
    <%= f.file_field :attachment_url %>
    <%= f.submit %>
  <% end %>
</div>

<script type="text/javascript">
  $(function() {
    var $form = $('form.direct_upload'),
      upload_url = '<%= escape_javascript(@presigned_post.url.to_s) %>',
      upload_form_data = '<%= escape_javascript(@presigned_post.fields.to_json.html_safe) %>';

    console.log('URL: ', upload_url);
    console.log('Form data: ', upload_form_data);

    if ($form.length) {
      $form.find('input[type=file]').each(function(index, input) {
        var $file_field = $(input);

        $file_field.fileupload({
          fileInput: $file_field,
          url: upload_url,
          type: 'POST',
          autoUpload: false,
          formData: upload_form_data,
          paramName: 'file',
          dataType: 'XML',

          add: function(e, data) {
            console.log('add callback fired.');
            $form.submit(function(e) {
              e.preventDefault();
              console.log('form submitted.');
              data.submit();
            });
          },
          start: function(e) {
            console.log('start callback fired');
          },
          done: function(e, data) {
            console.log('done callback fired');
          },
          fail: function(e, data) {
            console.log('fail callback fired');
            console.log(e);
            console.log(data);
          }
        });
      });
    }
  });
</script>

This is the response coming back from S3:

<Error>
  <Code>InvalidArgument</Code>
  <Message>Bucket POST must contain a field named 'key'.  If it is specified, please check the order of the fields.</Message>
  <ArgumentName>key</ArgumentName>
  <ArgumentValue></ArgumentValue>
  <RequestId>[filtered]</RequestId>
  <HostId>[filtered]</HostId>
</Error>

When the page loads you can see the expected output in the javascript console:

URL:  https://mybucket.s3.amazonaws.com/
Form data:  {"key":"attachments/d6313635-9735-4b84-9985-f9f62a036de8/${filename}","policy":"[FILTERED]","x-amz-credential":"[FILTERED]/us-east-1/s3/aws4_request","x-amz-algorithm":"AWS4-HMAC-SHA256","x-amz-date":"20150809T134239Z","x-amz-signature":"[FILTERED]"}

As you can see, there is a key field.

When you add a file to the file input field, the add callback fires and binds the submit action of the form, as expected. When the form is submitted, the request goes to S3, but the then the fail callback fires because a 400 is returned.

This question might be describing what the problem is but I have not been able to solve it based on the information provided.

The following is the request/response info, copied from Chrome dev tools.

Remote Address:54.231.17.17:443
Request URL:https://mybucket.s3.amazonaws.com/
Request Method:POST
Status Code:400 Bad Request

Response Headers
Access-Control-Allow-Methods:GET, POST, PUT
Access-Control-Allow-Origin:*
Connection:close
Content-Type:application/xml
Date:Sun, 09 Aug 2015 12:29:57 GMT
Server:AmazonS3
Transfer-Encoding:chunked
Vary:Origin, Access-Control-Request-Headers, Access-Control-Request-Method
x-amz-id-2:ymrt0MUlhf3bKqVWj+O5jhaUPXNEXy9HQh9PABmqzDkkb4Ods3Hy1LA++8G/Svri3LcOktpnGeE=
x-amz-request-id:545E755033D285F2

Request Headers
Accept:application/xml, text/xml, */*; q=0.01
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:331
Content-Type:multipart/form-data; boundary=----WebKitFormBoundary9vtTme67oAg1OMyL
Host:braidio.s3.amazonaws.com
Origin:http://localhost:3000
Referer:http://localhost:3000/things/new
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36

Request Payload
------WebKitFormBoundary9vtTme67oAg1OMyL
Content-Disposition: form-data; name="file"; filename="my_text.txt"
Content-Type: text/plain


------WebKitFormBoundary9vtTme67oAg1OMyL--

As you can see, the request payload, only contains the file, and not the key. It might be that the file needs to come after the all other fields in the post request and thats why S3 is not seeing the key field, as suggested in this answer .

Some relevant gems:

* jquery-rails (4.0.4)
* rails (4.2.3)
* aws-sdk (2.1.13)
* aws-sdk-core (2.1.13)
* aws-sdk-resources (2.1.13)

Also using jquery.fileupload.js 5.42.3

I'm not not sure how to get this working.

Thanks in advance!

Answer 1

I was able find the solution.

The form data the was generated on the backend that I captured on the frontend:

upload_form_data = '<%= escape_javascript(@presigned_post.fields.to_json.html_safe) %>'

had to be converted from a JSON string into a JavaScript object:

upload_form_data_obj = JSON.parse(upload_form_data);

Apparently, the $.fileupload function expects an object for formData, not a string.

With that change in place, the required form data is being included in the POST to S3 and it is successful.

Here is the working javascript code:

$(function() {
var $form = $('form.direct_upload'),
  upload_url,
  upload_form_data,
  upload_form_data_obj;

if ($form.length) {
  upload_url = '<%= escape_javascript(@presigned_post.url) %>'
  upload_form_data = '<%= escape_javascript(@presigned_post.fields.to_json.html_safe) %>';
  upload_form_data_obj = JSON.parse(upload_form_data);

  console.log('URL: ', upload_url);
  console.log('Form data: ', upload_form_data_obj);

  $form.find('input[type=file]').each(function(index, input) {
    var $file_field = $(input);
    $file_field.fileupload({
      fileInput: $file_field,
      url: upload_url,
      type: 'POST',
      autoUpload: false,
      formData: upload_form_data_obj, // needed to be an object, not a string
      paramName: 'file',
      dataType: 'JSON',

      add: function(e, data) {
        console.log('add callback fired.');
        $form.submit(function(e) {
          e.preventDefault();

          console.log('form submitted.');
          console.log(data);
          data.submit();
        });
      },
      start: function(e) {
        console.log('start callback fired');
      },
      done: function(e, data) {
        console.log('done callback fired');
      },
      fail: function(e, data) {
        console.log('fail callback fired');
        console.log(e);
        console.log(data);
      }
    });
  });
}
});