Asynchronous multiple files upload to AWS S3 bucket

Question

I want to upload multiple files to AWS S3 bucket with transfer manager from AWS Java SDK (aws-java-sdk-s3-1.11.66.jar), so my function over here:

public static void uploadDirectory(String virtualDirectoryKeyPrefix, File directory) {
    try {
        MultipleFileUpload multipleFileUpload = instance.transferManager.uploadFileList(
                instance.bucketName,
                virtualDirectoryKeyPrefix,
                directory,
                Arrays.asList(Objects.requireNonNull(directory.listFiles())));

        multipleFileUpload.waitForCompletion();
    } catch (InterruptedException e) {
        Logger.error("Not able to write to s3 bucket : " + virtualDirectoryKeyPrefix, e);
    }
}

The problem that it works only when I call the 'waitForCompletion' method and it makes my function synchronous because in this case, I wait until the operation finish. I expected that it must work without 'waitForCompletion' call, but it not, could U please help me? Thanks.

Answer 1

waitForCompletion only blocks until the transfer is complete( https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/Transfer.html#waitForCompletion-- ).

I suspect your transfer was working also before but because it is asynchronous you were not checking when it was done. In order to receive asynchronous updates add a ProgressListener as described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/HLTrackProgressMPUJava.html

Answer 2

Why can't you add a listener to the transferManager instance

    transferMangerInstance.upload(putObjectRequestInstance);
    transferMangerInstance.addProgressListener(ProgressListener progressListener);

    transferMangerInstance.addProgressListener(new ProgressListener() {
                @Override
                public void progressChanged(ProgressEvent progressEvent) {
                    if(progressEvent.getEventType() == ProgressEventType.TRANSFER_COMPLETED_EVENT) {
                  //Do your completion logic here
                    }
                }
            }