AWS SDK JS: Multipart upload to S3 resulting in Corrupt data

Question

Trying to upload an mp4 file using the AWS JS SDK initiating a multipart upload, I keep getting a file corrupt error when I try to download and play it on my local.

Gists of my code:

1. Initiating the multipart upload with params:

    const createMultipartUploadParams = { Bucket: bucketname, Key: fileHash.file_name, ContentType: 'video/mp4' // TODO: Change hardcode }; 

Call:

s3Instance.createMultipartUpload(createMultipartUploadParams, function(err, data) {
}

2. Doing the chunking: Params:

     const s3ChunkingParams = { chunkSize, noOfIterations, lastChunk , UploadId: data.UploadId } 

Reading the file:

     const reader = new FileReader();
     reader.readAsArrayBuffer(file)

Uploading each chunk:

        reader.onloadend = function onloadend(){
      console.log('onloadend');
      const partUploadParams = {
        Bucket: bucketname,
        Key: file_name,
        PartNumber: i, // Iterating over all parts
        UploadId: s3ChunkingParams.UploadId,
        Body: reader.result.slice(start, stop) // Chunking up the file
      };

      s3Instance.uploadPart(partUploadParams, function(err, data1) {
      }

3. Finally completing the multipartUpload:

   s3Instance.completeMultipartUpload(completeMultipartParams, function(err, data)

I am guessing the problem is how I am reading the file, so I have tried Content Encoding it to base64 but that makes the size unusually huge. Any help is greatly appreciated!

Tried this too

Answer 1

Only thing that could corrupt is perhaps you are uploading additionally padded content for your individual parts which basically leads to final object being wrong. I do not believe S3 is doing something fishy here.

You can verify after uploading the file what is the final size of the object, if it doesn't match with your local copy then you know you have a problem somewhere.

Are you trying to upload from browser?

Alternatively you can look at - https://github.com/minio/minio-js . It has minimal set of abstracted API's implementing most commonly used S3 calls.

Here is a nodejs example for streaming upload.

$ npm install minio
$ cat >> put-object.js << EOF

var Minio = require('minio')
var fs = require('fs')

// find out your s3 end point here:
// http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

var s3Client = new Minio({
  url: 'https://<your-s3-endpoint>',
  accessKey: 'YOUR-ACCESSKEYID',
  secretKey: 'YOUR-SECRETACCESSKEY'
})

var outFile = fs.createWriteStream('your_localfile.zip');
var fileStat = Fs.stat(file, function(e, stat) {
  if (e) {
    return console.log(e)
  }
  s3Client.putObject('mybucket', 'hello/remote_file.zip', 'application/octet-stream', stat.size, fileStream, function(e) {
    return console.log(e) // should be null
  })
})
EOF

putObject() here is a fully managed single function call for file sizes over 5MB it automatically does multipart internally. You can resume a failed upload as well and it will start from where its left off by verifying previously upload parts.

So you don't necessarily have to go through the trouble of writing lower level multipart calls.

Additionally this library is also isomorphic, can be used in browsers as well.