InputStream read part of file and output

Question

I have a set of large files ( about 2-3GB each) and I need to use InputStream to read it and output part of it to a new file.

//this is just for example
int size=100; 
String src="pathtofile";
OutputStream os = null; 
Inputstream is = new FileInputStream(new File(src));
byte[] buffer = new byte[size];
byte[] bufferis = getBytesFromIS(is); 

buffer=Arrays.copyOfRange(bufferis,0,buffer.length);

String tempstr=new String(buffer);  
byte[] tempBytes=Arrays.copyOfRange(bufferis, buffer.length,is.available());
os = new FileOutputStream(new File(dest));
copy(new ByteArrayInputStream(tempBytes), os); //function writing to file

This works fine with small files but when using on large files, I keep getting OutOfMemoryError , even when I set -Xmx6114m , still getting OutOfMemoryError .

Answer 1

The maximum array length is slightly less than Integer.MAX_VALUE , ie, about 2e9 . So you can't read the whole into a byte[] and have to use something else. A ByteByffer is probably the fastest solution (memory mapped file).

Answer 2

You don't show getBytesFromIS(is) but assuming that it reads the entire file, that's not necessary.

The easiest way is to use a library that already provides this functionality and is fully tested. For example, Apache Commons IOUtils .

Otherwise, you can use InputStream.read(b, off, len) , starting from an offset of 0 and with your desired length. Beware, however, that this function is not guaranteed to read the entire amount of data that you want.