AWS S3 Copy File Larger than 5GB (version 2.x)

Question

I'm trying to copy files from folder to folder in the same bucket in s3 storage. I need to copy files that are greater than 5GB and I saw in the docs that regular copy doesn't support this kind of copy see here

This link in the docs shows how to do it, but this code is for version 1.x and not 2.x.

I searched the new docs but I found only this and there is no code that shows how to multipart copy only regular copy.

It should be noted that another user asked about this but with no replies.

Answer 1

This code will help you to copy object with multipart in Java S3 SDK v2.

    private final S3Client s3Client = S3Client.builder().build();

    public void copyObjectWithMultiPart() {

        String destBucketName = "destination-bucket";
        String destObjectKey = "destination-object-key";
        String sourceBucketName = "source-bucket";
        String sourceObjectKey = "source-object-key";

        // Initiate the multipart upload.
        CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
                .bucket(destBucketName)
                .key(destObjectKey)
                .build();

        CreateMultipartUploadResponse multipartUploadResponse = s3Client.createMultipartUpload(createMultipartUploadRequest);

        // Get the object size to track the end of the copy operation.
        HeadObjectRequest headObjectRequest = HeadObjectRequest.builder()
                .bucket(sourceBucketName)
                .key(sourceObjectKey)
                .build();
        long objectSize = s3Client.headObject(headObjectRequest).contentLength();

        // Copy the object using 5 MB parts.
        long partSize = 5 * 1024 * 1024;
        long bytePosition = 0;
        int partNum = 1;
        List<CompletedPart> etags = new ArrayList<>();
        while (bytePosition < objectSize) {
            // The last part might be smaller than partSize, so check to make sure
            // that lastByte isn't beyond the end of the object.
            long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1);

            // Copy this part.
            UploadPartCopyRequest uploadPartCopyRequest = UploadPartCopyRequest.builder()
                    .sourceBucket(sourceBucketName)
                    .sourceKey(sourceObjectKey)
                    .destinationBucket(destBucketName)
                    .destinationKey(destObjectKey)
                    .uploadId(multipartUploadResponse.uploadId())
                    .partNumber(partNum)
                    .copySourceRange(String.format("bytes=%d-%d", bytePosition, lastByte))
                    .build();

            UploadPartCopyResponse uploadPartCopyResponse = s3Client.uploadPartCopy(uploadPartCopyRequest);
            etags.add(
                    CompletedPart.builder()
                            .partNumber(partNum++)
                            .eTag(uploadPartCopyResponse.copyPartResult().eTag())
                            .build()
            );
            bytePosition += partSize;
        }

        // Complete the upload request to concatenate all uploaded parts and make the copied object available.
        CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
                .parts(etags)
                .build();

        CompleteMultipartUploadRequest completeMultipartUploadRequest =
                CompleteMultipartUploadRequest.builder()
                        .bucket(destBucketName)
                        .key(destObjectKey)
                        .uploadId(multipartUploadResponse.uploadId())
                        .multipartUpload(completedMultipartUpload)
                        .build();

        s3Client.completeMultipartUpload(completeMultipartUploadRequest);
    }