在 Spring 引导中将 AWS S3 文件下载为 stream

Question

I want to expose an API to download a S3 bucket file content as stream to its consumers. The API URL is like /downloadfile/** which is GET request.

1. What should be my return type right now I tried with accept header= application/octet-stream which didn't work.
2. I don't want to write the content of file to any file and send it. It should be returned as a stream that's it.

Here is the controller pseudo code I wrote till now which is giving me 406 error all the time.

 @GetMapping(value = "/downloadfile/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
    public ResponseEntity<Object> downloadFile(HttpServletRequest request) {
       //reads the content from S3 bucket and returns a S3ObjectInputStream
       S3ObjectInputStream object = null;
       object = publishAmazonS3.getObject("12345bucket", "/logs/file1.log").getObjectContent();
       return object
    }

Any suggestions here on the way of doing this and what I am doing wrong?

Answer 1

I was able to download the file as a stream by using StreamingResponseBody class from Spring.

Here is the code I used:

    @GetMapping(value = "/downloadfile/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
    public ResponseEntity<S3ObjectInputStream> downloadFile(HttpServletRequest request) {
       //reads the content from S3 bucket and returns a S3ObjectInputStream
       S3Object object = publishAmazonS3.getObject("12345bucket", "/logs/file1.log");
       S3ObjectInputStream finalObject = object.getObjectContent();

        final StreamingResponseBody body = outputStream -> {
            int numberOfBytesToWrite = 0;
            byte[] data = new byte[1024];
            while ((numberOfBytesToWrite = finalObject.read(data, 0, data.length)) != -1) {
                System.out.println("Writing some bytes..");
                outputStream.write(data, 0, numberOfBytesToWrite);
            }
            finalObject.close();
        };
        return new ResponseEntity<>(body, HttpStatus.OK);
    }

The way to test the streaming is done correctly or not is to have a file of around 400mb to download. Reduce your Xmx to 256mb by passing the in the vm options. Now, compare the download functionality with and without using StreamingResponseBody , you will get OutofMemoryError when using the conventional OutputStreams for writing the content

Answer 2

You can solve with the below example

import java.io.ByteArrayOutputStream;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import com.grokonez.s3.services.S3Services;

@RestController
public class DownloadFileController {

    @Autowired
    S3Services s3Services;

    /*
     * Download Files
     */
    @GetMapping("/api/file/{keyname}")
    public ResponseEntity<byte[]> downloadFile(@PathVariable String keyname) {
        ByteArrayOutputStream downloadInputStream = s3Services.downloadFile(keyname);

        return ResponseEntity.ok().contentType(contentType(keyname))
                .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + keyname + "\"")
                .body(downloadInputStream.toByteArray());
    }

    private MediaType contentType(String keyname) {
        String[] arr = keyname.split("\\.");
        String type = arr[arr.length - 1];
        switch (type) {
        case "txt":
            return MediaType.TEXT_PLAIN;
        case "png":
            return MediaType.IMAGE_PNG;
        case "jpg":
            return MediaType.IMAGE_JPEG;
        default:
            return MediaType.APPLICATION_OCTET_STREAM;
        }
    }
}