Uploading large files with user metadata to Amazon S3 using java sdk

Question

I'm trying to use the Java API to upload large files, but I need to also add user metadata. If I just use

TransferManager tm = new TransferManager(new BasicAWSCredentials(accessKey, secretKey));

Upload upload = tm.upload(AmazonS3Manager.bucketName, imageKey, file);
upload.waitForCompletion();

Then everything works fine, but if I use:

ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(file.length());
metadata.addUserMetadata("My key", "My value");
FileInputStream input = new FileInputStream(file);
Upload upload = tm.upload(AmazonS3Manager.bucketName, imageKey, input, metadata);

Then it doesn't work, and I get the following output in the console:

Jul 5, 2011 4:33:15 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: POST https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploads: null, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, x-amz-meta-length: 312612077, )
Jul 5, 2011 4:33:16 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 2A5B3538795CE730
Jul 5, 2011 4:33:16 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, partNumber: 1, ) Headers: (Content-Length: 5242880, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 5E5AF291FBDBDD36
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, partNumber: 2, ) Headers: (Content-Length: 5242880, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:00 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: I/O exception (java.io.IOException) caught when processing request: Read error
Jul 5, 2011 4:34:00 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: Retrying request
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
WARNING: Unable to execute HTTP request: null
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: DELETE https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:01 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 204, AWS Request ID: 0EFC3F8D0FA6097E

Any help greatly appreciated!

Answer 1

You can use following code for your file upload:

    ObjectMetadata metadata = new ObjectMetadata();
        metadata.setContentLength(file.length());
        metadata.addUserMetadata("My key", "My value");
        FileInputStream input = new FileInputStream(file);
        PutObjectRequest putObjectRequest = new PutObjectRequest(AmazonS3Manager.bucketName, imageKey, input, metadata);
s3client.putObject(putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead));

Thanks

Answer 2

You may also need to set content type and, possibly, checksum as it can't be determined from input stream. This has some information for similar situation http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#putObject(com.amazonaws.services.s3.model.PutObjectRequest )

Answer 3

If you are trying to upload more than 5 GB file with updating metadata then amazon not allowing to do so. Because amazon s3 not providing copy operation for more then 5Gb file.

Amazon S3 recently started to support copy operation wtih multipart operation so for that i can not say you something yet.

https://forums.aws.amazon.com/thread.jspa?messageID=256605

Thanks