How to copy files from S3 bucket from one region to another region using aws java sdk?

Question

We have two regions in AWS, where there is a AWS S3 Bucket in each region. How do i copy the files inside the bucket from one region to another using AWS Java SDK?

We do not have access to credentials of the source region bucket, but We have a presigned URL for the source of each file in the source region bucket, using which we can download the file and then use the AWS Upload URL to upload it to destination region bucket.

There are space constraints while downloading the file, so we are trying to find a way to copy files from bucket in one region to another using AWS Java SDK. is this achievable?

Edit:

For some more clarity, both buckets are already created, and it is a continuous process to be implemented as part of our code. It is not a one time activity.

Answer 1

Normally, copying between buckets is easy. The CopyObject() command can copy objects between buckets (even buckets in different regions and different accounts) without having to download/upload the files.

However, since you only have access to the files via a pre-signed URL, you will need to:

• Download each file individually
• Upload each file to the target Amazon S3 bucket using the PutObject() command in the AWS SDK for Java

This would best be done on an Amazon EC2 instance in either the source or destination regions.

It would be much better if the owner of the source bucket could provide you with "normal" access to the objects so that you can use CopyObject() , or if they were able to copy the objects to your bucket for you.

Answer 2

I would recommend exploring the S3 cross-region replication service first.

Programmatically copying files from bucket to bucket has a lot to concern:

• S3 API rate limit
• AWS cost to transfer data
• AWS cost to host your solution, be it hosted in EC2, container, or serverless
• cost to develop and maintain your codebase
• what about S3 object metadata? does metadata need to be preserved?
• what about S3 buckets with versioning enabled?

Answer 3

the below code copies the file from one region to another region. If the two different region shares same access and secret key

System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY, "true");

    AmazonS3 s3ClientBuilder = null;
    AmazonS3 s3desClientBuilder = null;
    TransferManager transferManager = null;
    try {

        ClientConfiguration clientCfg = new ClientConfiguration();
        clientCfg.setProtocol(Protocol.HTTPS);
        clientCfg.setSignerOverride("S3SignerType");
        AWSStaticCredentialsProvider credentialProvidor = new AWSStaticCredentialsProvider(
                new BasicAWSCredentials(accessKey, secretKey));

        s3ClientBuilder = AmazonS3ClientBuilder//
                .standard()//
                .withCredentials(credentialProvidor)//
                .withEndpointConfiguration(new EndpointConfiguration(s3Endpoint, region.getName()))//
                .withClientConfiguration(clientCfg)//
                .build();

//list all bucket names in the s3region

        List<Bucket> buckets = s3ClientBuilder.listBuckets();
        System.out.println("Your Amazon S3 buckets are:");
        for (Bucket b : buckets) {
            System.out.println("* " + b.getName());
        }

//List object and objectkey inside the bucket

        ListObjectsRequest lor = new ListObjectsRequest()
                .withBucketName(SOURCE_BUCKET_NAME)
                .withPrefix("vivek/20210801");
        ObjectListing objectListing = s3ClientBuilder.listObjects(lor);
        for (S3ObjectSummary summary: objectListing.getObjectSummaries()) {

            SOURCE_KEY=summary.getKey();
            DESTINATION_KEY=SOURCE_KEY

            s3desClientBuilder = AmazonS3ClientBuilder//
                    .standard()//
                    .withCredentials(credentialProvidor)//
                    .withEndpointConfiguration(new EndpointConfiguration(s3desEndpoint, region.getName()))//
                    .withClientConfiguration(clientCfg)//
                    .build();

             transferManager = TransferManagerBuilder.standard()
                    .withS3Client(s3desClientBuilder)
                    .build();

            Copy copy = transferManager.copy(new CopyObjectRequest(SOURCE_BUCKET_NAME, SOURCE_KEY,
                    DESTINATION_BUCKET_NAME, DESTINATION_KEY),
                    s3ClientBuilder, null);
            copy.waitForCopyResult();   
        }
        transferManager.shutdownNow();
        s3ClientBuilder.shutdown();
        s3desClientBuilder.shutdown();