spring boot how to download all files in a folder of s3 to a local directory?

Question

I am new to spring, I want to use TransferManager to download all files in a certain folder, and I also want to know how should I get users to get their local path when downloading files on webpage. After I run the code, the code does not report an error, but the local path I wrote also has no s3 file, below is my code:

  @RequestMapping(method = RequestMethod.GET, path = "/{id}/download")
    void downloadGroup(
        @PathVariable("id") @NotNull Integer id
    ) throws IOException {
        String bucketName;
        String keyPrefix;
        String destinationDirectory = "Users/Download";
        AWSCredentials credentials = new BasicAWSCredentials(
            awsAccessKeyId, awsSecretAcessKey);

        AmazonS3 s3 = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(credentials))
            .withEndpointConfiguration(
                new AwsClientBuilder.EndpointConfiguration(
                    awsServiceEndpoint,
                    awsDefaultRegion))
            .build();

        ObjectListing objects = s3.listObjects(bucketName, keyPrefix);

        GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, project);

        TransferManager tm = TransferManagerBuilder.standard()
            .withS3Client(s3)
            .build();

//        Download download = tm.download(bucketName, keyPrefix, new File(destinationDirectory));
        try {
            MultipleFileDownload download = tm.downloadDirectory(
                bucketName,
                keyPrefix,
                new File(destinationDirectory));
            download.waitForCompletion();
            LOG.info("Download complete.");
        } catch (AmazonClientException amazonClientException) {
            LOG.info("Unable to download file, download was aborted.");
            throw new RuntimeException(amazonClientException);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        tm.shutdownNow();
    }

Below is my newly modified code:

@RequestMapping(method = RequestMethod.GET, path = "/{id}/download")
public ResponseEntity<ByteArrayResource> downloadGroup(
        @PathVariable("id") @NotNull Integer id
    ) throws IOException {
        String bucketName;
        String keyPrefix;

        AWSCredentials credentials = new BasicAWSCredentials(
            awsAccessKeyId, awsSecretAcessKey);

        AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(credentials))
            .withEndpointConfiguration(
                new AwsClientBuilder.EndpointConfiguration(
                    awsServiceEndpoint,
                    awsDefaultRegion))
            .build();
        byte[] data = null;
        String fileName = null;
        ObjectListing objects = s3Client.listObjects(bucketName, keyPrefix);
        List<S3ObjectSummary> objectSummaries = objects.getObjectSummaries();
        for (S3ObjectSummary objectSummary : objectSummaries) {
            if (objectSummary.getKey().contains(".csv")) {
                data = downloadFile(s3Client, bucketName, objectSummary.getKey());
                fileName = URLEncoder.encode(objectSummary.getKey(), "UTF-8").replaceAll("\\+", "%20");
                System.out.println(fileName);
            }
        }
        HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        httpHeaders.setContentLength(data.length);
        httpHeaders.setContentDispositionFormData("attachment", fileName);
        ByteArrayResource resource = new ByteArrayResource(data);
        return new ResponseEntity<>(resource, httpHeaders, HttpStatus.OK);
    }
public byte[] downloadFile(AmazonS3 s3, String bucketName, String keyName) {
        byte[] content = null;
        S3Object s3Object = s3.getObject(bucketName, keyName);
        S3ObjectInputStream stream = s3Object.getObjectContent();
        try {
            content = IOUtils.toByteArray(stream);
            s3Object.close();
        } catch(final IOException ex) {
            LOG.info("IO Error Message= " + ex.getMessage());
        }
        return content;
    }

Answer 1

When reading the origin example from https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-transfermanager.html#tranfermanager-download-directory and reading your code and your wanted behavior above, I think your problem is, that you store the s3 content at the spring boot application side.

For example: When you started your spring boot application at

/opt/myserver/MyServer.jar

using your code will download your s3 bucket content data into /opt/myserver/Users/Download

But your wanted behavior is to provide the files to the users on browser side - or did I miss it?

I would try to fetch the s3 content data from your spring boot app and return a Zip OutputStream to waiting client side (browser) (maybe you will have to download temporary files, zip them, delete temp content etc.)