如何将大文件下载到字节数组中

Question

I am trying to download large file using spring Rest Template:

    private Result download(Long id) throws Exception {
    Result result = new Result();
    try {
        RestTemplate restTemplate = restTemplateFactory.getRestTemplate();
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        headers.set(AUTHORIZATION, authorizationBL.getAccessToken().getAccessToken());
        HttpEntity<MultiValueMap<String, String>> request = new HttpEntity<MultiValueMap<String, String>>(null, headers);
        Map<String, Long> valuesMap = new HashMap<String, Long>(1);
        valuesMap.put(ID, id);
        ResponseEntity<byte[]> response = restTemplate.exchange(downloadURL, HttpMethod.GET, request, byte[].class);
        byte[] stream = response.getBody();
        result.setData(stream);
        result.setStatus(Result.Status.SUCCESS);
    } catch (Exception e) {
        throw e;
    }
    return result;
}

when file size is large lets say above 100 MB then I am getting issue:

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_60]
at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191) ~[na:1.8.0_60]
at org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:59) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:38) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:193) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:105) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:924) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:908) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:662) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:620) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:538) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

I increased heap size upto 10g but it also didn't help.

Answer 1

Instead of downloading the file into a byte array, save it to disk. Then you can read it from wherever else you need to in your program. For example, changing your method to

/**
 * returns a File corresponding to the tmp file download for this resource
 */
private File download(Long id) throws IOException {
    File tmp = null;
    try {
        RestTemplate restTemplate = restTemplateFactory.getRestTemplate();
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        headers.set(AUTHORIZATION, authorizationBL.getAccessToken().getAccessToken());
        HttpEntity<MultiValueMap<String, String>> request = new HttpEntity<MultiValueMap<String, String>>(null, headers);
        ResponseEntity<byte[]> response = restTemplate.exchange(downloadURL, HttpMethod.GET, request, byte[].class);

        if (response.getStatusCode() == HttpStatus.OK) {
            tmp = Files.createTempFile("yourprefix", "yoursuffix");
            Files.write(tmp.toPath(), response.getBody());
        } else {
            // TODO handle this case
        }
    } catch (Exception e) {
        throw new IOException("An exception occurred while downloading", e);
    }
    return tmp;
}

A few observations:

1. The valuesMap wasn't used anywhere (also removed)
2. result also removed

Answer 2

我能够使用流而不是将文件存储在 byte[] 中并使用IOUtils.copyLarge(inputStream, outputStream)来解决这个问题。