Problems when uploading large files to Amazon S3

Question

I tried to use Amazon-SDK(Java) sample code S3TransferProgressSample.java to upload large files to Amazon-S3 storage ( also posted here on AWS docs ).

But when I am trying to upload 11 GB files, the upload is getting stuck at different points with the error message:

Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot). 

It looks like that after IOException occurs SDK is not able to retry the request (see below).

Does anyone encounter this? What is the best-practice to resolve this? Any code is appreciated.

 INFO: Received successful response: 200, AWS Request ID:
 2B66E7669E24DA75<br> Jan 15, 2011 6:44:46 AM
 com.amazonaws.http.HttpClient execute<br> INFO: Sending Request: PUT
 s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
 (uploadId:
     m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )<br> Jan 15, 2011 6:45:10 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
     Jan 15, 2011 6:45:10 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     INFO: Retrying request<br> Jan 15, 2011 6:45:12 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:12 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
     Jan 15, 2011 6:45:12 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
     Jan 15, 2011 6:45:13 AM
     org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
     INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
     com.amazonaws.http.HttpClient execute<br>
     **WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.**<br> Jan 15, 2011 6:45:14 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:14 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:16 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:16 AM

 com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:17 AM
     com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
     HTTP request: Unbuffered entity enclosing request can not be
     repeated.<br> Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient
     execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
     enclosing request can not be repeated.<br> Jan 15, 2011 6:45:19 AM
     com.amazonaws.http.HttpClient execute<br> ....<br> Jan 15, 2011
     6:45:21 AM com.amazonaws.http.HttpClient handleResponse<br>
     **INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007**<br> Jan 15, 2011 6:45:21 AM
     com.amazonaws.http.HttpClient execute<br> ...<br> Jan 15, 2011 6:45:19
     AM com.amazonaws.http.HttpClient execute<br> INFO: Sending Request:
     DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt
     Parameters:<br> ...<br> Jan 15, 2011 6:47:01 AM
     com.amazonaws.http.HttpClient handleErrorResponse<br> INFO: Received
     error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595,
     AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload
     does not exist. The upload ID may be invalid, or the upload may have
     been aborted or completed.<br>

Answer 1

Try using the low level API .

This will give you far more control when things go wrong, as they are likely to do with an 11GB file.

Requests to and from S3 do fail from time to time. With the low level API, you'll be able to retry a part of the upload if it fails.

Refactoring the example in the Amazon docs a bit:

// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
    // Last part can be less than 5 MB. Adjust part size.
    partSize = Math.min(partSize, (contentLength - filePosition));

    // Create request to upload a part.
    UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(existingBucketName).withKey(keyName)
                .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                .withFileOffset(filePosition)
                .withFile(file)
                .withPartSize(partSize);

    // repeat the upload until it succeeds.
    boolean anotherPass;  
        do {
              anotherPass = false;  // assume everythings ok
              try {
                  // Upload part and add response to our list.
                  partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
              } catch (Exception e) {
                    anotherPass = true; // repeat
              }
        } while (anotherPass);

     filePosition += partSize;
}

   // Step 3: complete.
   CompleteMultipartUploadRequest compRequest = new 
                     CompleteMultipartUploadRequest(
                                existingBucketName, 
                                keyName, 
                                initResponse.getUploadId(), 
                                partETags);

   s3Client.completeMultipartUpload(compRequest);

Note: I am not a java developer so I could have messed things up syntactically, but hopefully this gets you going in the right direction. Also, you'll want to add in a 'retry counter' to prevent an endless loop if the upload repeatedly fails.

Answer 2

附带说明一下，如果您尝试对已在分段上传下的密钥进行分段上传，则可能会引发 404 错误。

Answer 3

I think you should try Multipart API supported by AWS.

Check this out : http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html

Answer 4

I wanted to add a comment to Geoff Appleford's answer but SO wouldn't allow me to. In general his answer to use low level API works fine but even if we do now have a do-while loop the way for loop is designed there is in-built retry logic. In his code snippet the file position increases only when there is a success otherwise you are uploading the same part again.

Answer 5

The answer of Geoff Appleford works for me. However, I would add a && retryCount < MAX_RETRIES to the while loop control statement and increment of the retryCount on every exception caught inside the while.

Aviad