I am facing an issue while downloading a large json file from internet (20+GB), specifically how it's being saved to the local disk. I don't want to parse the json, just download it.
There are several similar questions answered in SO, but i haven't seen people have issues with the output - I am not sure what I'm missing.
Anyways, when I execute the following code, it downloads the json and puts the corresponding byte array into file, and not save it in string format.
OkHttpClient httpClient = new OkHttpClient.Builder()
.connectTimeout(CONNECT_TIMEOUT, TimeUnit.MINUTES)
.readTimeout(CONNECT_TIMEOUT, TimeUnit.MINUTES)
.writeTimeout(CONNECT_TIMEOUT, TimeUnit.MINUTES)
.build();
Request request = new Request.Builder()
.url(path)
.header("User-Agent", ClientConstants.USER_AGENT)
.addHeader("Accept-Encoding", "gzip, deflate, sdch, br")
.addHeader("Content-Type", "application/json")
.build();
Response response = httpClient.newCall(request).execute();
if (!response.isSuccessful()) {
log.error("failed download for url : {0}, with status code {1}", url, response.code());
if (response.body() != null) {
response.body().close();
}
throw new IOException("failed download for url=" + url);
}
else {
try(BufferedInputStream inputStream = new BufferedInputStream(response.body().byteStream())){
byte[] readBuffer = new byte[10000];
FileOutputStream outputStream = new FileOutputStream(this.downloadFilePath);
int bytesRead;
while ((bytesRead = inputStream.read(readBuffer, 0, 10000)) != -1) {
outputStream.write(readBuffer, 0, bytesRead);
outputStream.flush();
}
outputStream.close();
} catch (IOException e) {
log.error("failed download for url=" + url, e);
throw e;
}
}
The code works. I just had to remove the incorrect header as Victor Gubin suggested.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.