[英]How do I download a large json file from internet without parsing it (java, okhttp3)?
从互联网(20 + GB)下载较大的json文件时,尤其是如何将其保存到本地磁盘时,我遇到了一个问题。 我不想解析json,只需下载它即可。
在SO中也回答了几个类似的问题,但是我还没有看到人们的输出出现问题-我不确定我缺少什么。
无论如何,当我执行以下代码时,它会下载json并将相应的字节数组放入文件中,而不是以字符串格式保存它。
OkHttpClient httpClient = new OkHttpClient.Builder()
.connectTimeout(CONNECT_TIMEOUT, TimeUnit.MINUTES)
.readTimeout(CONNECT_TIMEOUT, TimeUnit.MINUTES)
.writeTimeout(CONNECT_TIMEOUT, TimeUnit.MINUTES)
.build();
Request request = new Request.Builder()
.url(path)
.header("User-Agent", ClientConstants.USER_AGENT)
.addHeader("Accept-Encoding", "gzip, deflate, sdch, br")
.addHeader("Content-Type", "application/json")
.build();
Response response = httpClient.newCall(request).execute();
if (!response.isSuccessful()) {
log.error("failed download for url : {0}, with status code {1}", url, response.code());
if (response.body() != null) {
response.body().close();
}
throw new IOException("failed download for url=" + url);
}
else {
try(BufferedInputStream inputStream = new BufferedInputStream(response.body().byteStream())){
byte[] readBuffer = new byte[10000];
FileOutputStream outputStream = new FileOutputStream(this.downloadFilePath);
int bytesRead;
while ((bytesRead = inputStream.read(readBuffer, 0, 10000)) != -1) {
outputStream.write(readBuffer, 0, bytesRead);
outputStream.flush();
}
outputStream.close();
} catch (IOException e) {
log.error("failed download for url=" + url, e);
throw e;
}
}
该代码有效。 我只需要按照Victor Gubin的建议删除不正确的标头。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.