从HDFS下载大文件

Question

I was given DataInputStream from a HDFS client for a large file (around 2GB) and I need to store it as a file on my host.

I was thinking about using apache common IOUtils and doing something like this...

File temp = getTempFile(localPath);
DataInputStream dis = HDFSClient.open(filepath); // around 2GB file (zipped)
in = new BufferedInputStream(dis);
out = new FileOutputStream(temp);
IOUtils.copy(in, out);

I was looking for other solutions that can work better than this approach. Major concern for this is to use buffering in both input and IOUtils.copy ...

Answer 1

For files larger than 2GB is recommended to use IOUtils.copyLarge() (if we are speaking about the same IOUtils: org.apache.commons.io.IOUtils )

The copy in IOUtils uses a default buffer size of 4Kb (although you can specify another buffer size as a parameter).

The difference between copy() and copyLarge() is the returning result.

For copy() , if the stream is bigger than 2GB you will succeed with the copy but the result is -1.

For copyLarge() the result is exactly the amount of bytes you copied.

See more in the documentation here: http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/IOUtils.html#copyLarge(java.io.InputStream,%20java.io.OutputStream)