使用aws js sdk，无需使用cognito，即可将浏览器中的大文件（> 10GB）直接从浏览器安全地分段上传到s3

Question

Objective : to use aws js sdk and do multipart upload for large files(>10GB) directly from browser to s3 in a secure way.

Achieved : successfully able to upload files about 15GB directly from browser to s3 bucket.

Issue : without using amazon cognito and hard-coding the keys and secret do the same.

From what we I have read form docs and what aws support team has replied that I can achieve this with STS but I did try that out check this :

from boto.s3.connection import S3Connection
from boto.sts import STSConnection
sts = STSConnection('our_key', 'our_secret')
user = sts.get_federation_token('guest_user_1')
user= sts.assume_role(role_arn='arn:aws:iam::008557872112:role/Cognito_testAuth_Role', role_session_name='Cognito_testAuth_Role_temp')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/aameer/.virtualenvs/indeedev/lib/python3.4/site-packages/boto/sts/connection.py", line 384, in assume_role
    return self.get_object('AssumeRole', params, AssumedRole, verb='POST')
  File "/home/aameer/.virtualenvs/indeedev/lib/python3.4/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 403 Forbidden
<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>AccessDenied</Code>
    <Message>Roles may not be assumed by root accounts.</Message>
  </Error>
  <RequestId>95e7efc9-12b3-11e6-b1e2-ffa0432dfc4b</RequestId>
</ErrorResponse>

We had read that the roles may not be assumed by root accounts but we couldn't figure out how to create these roles then. What we are looking for is an example which would explain the full process of: a) create a non root user to be used in the above case ie How to create temp role (whether on back-end with the help of boto or on front-end with aws js sdk) b)and then use it on front-end to achieve our objective.

as the docs are highly confusing on this.

Note that : we are using django 1.7 and python3.4 with boto==1.3.1 and aws js sdk 2.2.13 to achieve this.

I am also sharing the js code

//the uploads(from browser to s3) had issues with large videos(videos greater than 5GB) , the earlier implementation gist is as under:

var xhr = new XMLHttpRequest();

sign = data['signature'] //from backend
policy_json = data['policy'] //from backend

console.log('values from backend')
console.log(sign,policy_json)
var fd = new FormData();
fd.append('key', key);
fd.append('acl', 'public-read'); 
fd.append('AWSAccessKeyId', 'our_key');
fd.append('policy', policy_json);
fd.append('signature',sign);
fd.append("file",file);
xhr.upload.addEventListener("progress", $scope.uploadProgress, false);
xhr.addEventListener("load", $scope.uploadComplete, false);
xhr.addEventListener("error", uploadFailed, false);
xhr.addEventListener("abort", uploadCanceled, false);
if (window.location.href.indexOf('8000') > -1 || window.location.href.indexOf('preproduction') > -1){
   xhr.open('POST', 'https://bucket1.s3.amazonaws.com/', true); //MUST BE LAST LINE BEFORE YOU SEND 
}else{
   xhr.open('POST', 'https://bucket2.s3.amazonaws.com/', true); //MUST BE LAST LINE BEFORE YOU SEND 
}
xhr.setRequestHeader("enctype", "multipart/form-data");
//xhr.setRequestHeader("Content-Type", "undefined");
xhr.send(fd);
if (window.location.href.indexOf('8000') > -1 || window.location.href.indexOf('preproduction') > -1){
   s3URL='https://bucket1.s3.amazonaws.com/'+projectid+'/video/'+projectid+'_'+$scope.video_id+'.'+file_extension;
}else{
   s3URL='https://bucket2.s3.amazonaws.com/'+projectid+'/video/'+projectid+'_'+$scope.video_id+'.'+file_extension;
}


// to make larger files (theoritally upto 5TB) we used aws's js sdk to enable the multipart upload and new implementation which is working
//(for now whatever we threw at it(about 15GB) it uploaded it successfully so the code has improved the robustness , time and size of uploads


//But the thing is we dont want the aws key and secret to be on the front end as mentioned in the following snippet:

AWS.config.update({accessKeyId: 'our_key', secretAccessKey: 'our-secret'});

//based on the server we use the bucket
if (window.location.href.indexOf('8000') > -1 || window.location.href.indexOf('preproduction') > -1){
   AWS.config.region = 'ap-southeast-1';
   bucket_name = 'bucket1'
}else{
   AWS.config.region = 'us-west-1';
   bucket_name = 'bucket2'
}

// Upload the File
var bucket = new AWS.S3({params: {Bucket: bucket_name}});
var params = {Key: key,
            Body: file,
            acl:'public-read'}
var upload_carrier = bucket.upload(params)

upload_carrier.on('httpUploadProgress', function(evt) {
   console.log(evt.loaded *100/evt.total)
   var percentComplete = Math.round(evt.loaded * 100 / evt.total);
   document.getElementById('progressNumber').innerHTML = percentComplete.toString() + '% Complete';
   $(".progress-bar").width(percentComplete.toString() + '%');
})

upload_carrier.send(function(err, data) {
  s3URL='https://bucket1.s3.amazonaws.com/'+projectid+'/video/'+projectid+'_'+$scope.video_id+'.'+file_extension;
  toastr.success("Video was uploaded successfully");
  _kmq.push(['record', 'Upload Finish', {'Upload Type':'Browser'}]);
  $scope.addVideo(); // adds a video on our backend
  console.log(err, data)
})

$scope.cancelUpload= function(evt) {
    toastr.error("The upload has been canceled by the user or the browser dropped the connection.");
    $scope.uploadchange=false;
    $scope.uploadbtn=true;
    $scope.isSaving=false;
    location.reload()
}

//from the docs it appears that we can use amazon incognito where we can use one of the authentication providers like fb, google amazon etc 
//to authenticate users but we don't want another level of authentication, we even tried to use amazon identity pool but it didn't work for some reason 
//and moreover it still doesn't solve the issue of not having keys and secret on front-end as we have to use something like this if we are not wrong

Set the region where your identity pool exists (us-east-1, eu-west-1)
AWS.config.region = 'us-east-1';
// Configure the credentials provider to use your identity pool
AWS.config.credentials = new AWS.CognitoIdentityCredentials({
    IdentityPoolId: 'identity-pool-id',
});

// Make the call to obtain credentials
AWS.config.credentials.get(function(){
    // Credentials will be available when this function is called.
    var accessKeyId = AWS.config.credentials.accessKeyId;
    var secretAccessKey = AWS.config.credentials.secretAccessKey;
    var sessionToken = AWS.config.credentials.sessionToken;
});

//we are trying to avoid custom authentication in cognito as it feels too much of work and again cant see clear examples.
// we also know that we can use the js SDK to assume a role temporarily that would allow uploading to S3
//This would mean that we do not have to use Cognito and also do not need to hard code keys. 
//These credentials by default are live for an hour. and if they only allow access to upload to S3 there is very little risk to our other resources.

// and that is what had tried above with boto but still not luck We have been stuck on this issue for way too long,I would highly appreciate some code snippets which could help

some of the links which we had consulted: https://docs.aws.amazon.com/cognito/latest/developerguide/developer-authenticated-identities.html \\ How to give permission to a federated user in boto to an s3 bucket? \\ https://www.whitneyindustries.com/aws/2014/11/16/boto-plus-s3-plus-sts-tokens.html \\ http://boto.cloudhackers.com/en/latest/ref/sts.html \\ http://boto.cloudhackers.com/en/latest/ref/sts.html#id7 \\ http://boto.cloudhackers.com/en/latest/ref/sts.html#id19 \\ http://boto.cloudhackers.com/en/latest/ref/sts.html \\ http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html \\ http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-api.html \\

thanks,

Answer 1

 <Message>Roles may not be assumed by root accounts.</Message>

You shouldn't be using your root account credentials for anything . You need to be using an IAM user. Everywhere that you are using root credentials now, those need to be replaced with an IAM user's credentials.

See https://aws.amazon.com/iam/details/manage-users/

See also Throw Away the Password to your AWS Account by Eric Hammond.