Store Java object to Amazon S3 bucket

Question

I have a Java object. I get this as a response from an underlying API of our system. I need to store this object in a folder inside an S3 bucket.

I found this https://stackoverflow.com/a/41017427/9715520 . However, as I said I need to upload the object in a folder inside the bucket. Is there a way I can specify the folder name as well. The example tells about saving the object to an S3 bucket. What if there are multiple folders within a bucket and one needs to store in a specific folder. Do we have something for that?

This is how I am doing it. Also is the below code sufficient enough or do I need to create s3 client explicitly to upload the object?

private void s3Upload(MyObject myObject) {
         
         TransferManager transferManager = TransferManagerBuilder.standard().build();
         
         ObjectMapper objectMapper = new ObjectMapper(); 
         byte[] bytesToWrite = objectMapper.writeValueAsBytes(myObject);

         ObjectMetadata omd = new ObjectMetadata();
         omd.setContentLength(bytesToWrite.length);
         String uniqueID = UUID.randomUUID().toString();
         transferManager.upload(<bucketName>, uniqueID, new ByteArrayInputStream(bytesToWrite), omd);
    }

Answer 1

To upload an object (a PDF for example), its recommended that you use AWS SDK for Java V2 . Using this API, you have 2 choices.

1 . You can upload an object to an Amazon S3 bucket using the S3Client object.

2 . You can upload an object to an Amazon S3 bucket using the TransferManager object.

Amazon S3 console supports the folder concept as a means of grouping objects. For example, if you want to store a PDF file in a folder named folder2 , specify the name in the objectKey . For example:

String objectKey = "folder2/myPDF.pdf"

When you ran this code, myPDF is in folder2 -- as shown in this screen shot:

If you are not familiar with using the AWS SDK for Java V2 , refer to the Java V2 DEV Guide for all the concepts you need. For example, how to work with Credentials.

S3Client

The following code shows you how to use this object to upload content.

package com.example.s3;

// snippet-start:[s3.java2.s3_object_upload.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.S3Exception;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
// snippet-end:[s3.java2.s3_object_upload.import]

/**
 * Before running this Java V2 code example, set up your development environment, including your credentials.
 *
 * For more information, see the following documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 */

public class PutObject {

    public static void main(String[] args) {
        final String usage = "\n" +
                "Usage:\n" +
                "  <bucketName> <objectKey> <objectPath> \n\n" +
                "Where:\n" +
                "  bucketName - The Amazon S3 bucket to upload an object into.\n" +
                "  objectKey - The object to upload (for example, book.pdf).\n" +
                "  objectPath - The path where the file is located (for example, C:/AWS/book2.pdf). \n\n" ;

        if (args.length != 3) {
            System.out.println(usage);
            System.exit(1);
        }

        String bucketName =args[0];
        String objectKey = args[1];
        String objectPath = args[2];
        System.out.println("Putting object " + objectKey +" into bucket "+bucketName);
        System.out.println("  in bucket: " + bucketName);

        ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
        Region region = Region.US_EAST_1;
        S3Client s3 = S3Client.builder()
                .region(region)
                .credentialsProvider(credentialsProvider)
                .build();

        String result = putS3Object(s3, bucketName, objectKey, objectPath);
        System.out.println("Tag information: "+result);
        s3.close();
    }

    // snippet-start:[s3.java2.s3_object_upload.main]
    public static String putS3Object(S3Client s3,
                                     String bucketName,
                                     String objectKey,
                                     String objectPath) {

        try {

            Map<String, String> metadata = new HashMap<>();
            metadata.put("x-amz-meta-myVal", "test");

            PutObjectRequest putOb = PutObjectRequest.builder()
                    .bucket(bucketName)
                    .key(objectKey)
                    .metadata(metadata)
                    .build();

            PutObjectResponse response = s3.putObject(putOb,
                    RequestBody.fromBytes(getObjectFile(objectPath)));

           return response.eTag();

        } catch (S3Exception e) {
            System.err.println(e.getMessage());
            System.exit(1);
        }
        return "";
    }

    // Return a byte array.
    private static byte[] getObjectFile(String filePath) {

        FileInputStream fileInputStream = null;
        byte[] bytesArray = null;

        try {
            File file = new File(filePath);
            bytesArray = new byte[(int) file.length()];
            fileInputStream = new FileInputStream(file);
            fileInputStream.read(bytesArray);

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (fileInputStream != null) {
                try {
                    fileInputStream.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
        return bytesArray;
    }
    // snippet-end:[s3.java2.s3_object_upload.main]
}

TransferManager object

The following code shows you how to use this Java object to upload content to an S3 bucket.

package com.example.transfermanager;

import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.transfer.s3.FileUpload;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import java.nio.file.Paths;

/**
 * Before running this Java V2 code example, set up your development environment, including your credentials.
 *
 * For more information, see the following documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 */
public class UploadObject {

    public static void main(String[] args) {

        final String usage = "\n" +
                "Usage:\n" +
                "  <bucketName> <objectKey> <objectPath> \n\n" +
                "Where:\n" +
                "  bucketName - The Amazon S3 bucket to upload an object into.\n" +
                "  objectKey - The object to upload (for example, book.pdf).\n" +
                "  objectPath - The path where the file is located (for example, C:/AWS/book2.pdf). \n\n" ;

       if (args.length != 3) {
            System.out.println(usage);
            System.exit(1);
       }

        long mb = 1024;
        String bucketName = args[0];
        String objectKey = args[1];
        String objectPath = args[2];


        System.out.println("Putting an object into bucket "+bucketName +" using the S3TransferManager");
        ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
        Region region = Region.US_EAST_1;
        S3TransferManager transferManager = S3TransferManager.builder()
                .s3ClientConfiguration(cfg ->cfg.region(region)
                        .credentialsProvider(credentialsProvider)
                        .targetThroughputInGbps(20.0)
                        .minimumPartSizeInBytes(10 * mb))
                .build();

        uploadObjectTM(transferManager, bucketName, objectKey, objectPath);
        System.out.println("Object was successfully uploaded using the Transfer Manager.");
        transferManager.close();
    }

    public static void uploadObjectTM( S3TransferManager transferManager, String bucketName, String objectKey, String objectPath) {

        FileUpload upload =
                transferManager.uploadFile(u -> u.source(Paths.get(objectPath))
                        .putObjectRequest(p -> p.bucket(bucketName).key(objectKey)));
        upload.completionFuture().join();
    }
}

Another Update

Look at this line of code:

PutObjectResponse response = s3.putObject(putOb, RequestBody.fromBytes (getObjectFile(objectPath)));

As long as you get your data you want to upload to a byte array, you can upload it to Amazon S3 using this code. I used a physical PDF file as an example. But if you get the byte[] from another source it still works.

The RequestBody supports other methods too as discussed in these Javadocs.

https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/sync/RequestBody.html

Hope this clears up your doubt.