How to read a file as byte[] in Java when it is bigger than your RAM?

Question

I have written a Java program which encrypts a file using my custom algorithm. It reads a file as byte[] and the my function modifies this array. Finally, this array is saved as encrypted file. But the problem is that if I read a file which is greater than my RAM, it no longer can encrypt. So what's the solution?

Edit: For now assume that there is no encryption. I just have to reverse the file.

Answer 1

Instead of reading the entire thing to a buffer and then writing it all out at once in encrypted form, you can use streams to read and write chunks at a time. Specifically, you could use a CipherOutputStream to encrypt as you go.

Just as an example of the kind of thing you might do:

byte[] buffer = new byte[4096];
FileInputStream fileInStream = new FileInputStream("in.txt");
FileOutputStream fileStream = new FileOutputStream("test.bin");
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
CipherOutputStream cipherStream = new CipherOutputStream(fileStream, cipher);

while(fileInStream.read(buffer) > 0){
    cipherStream.write(buffer);
}

If you are just trying to reverse the entire file without reading the whole thing into memory all at once, you could do something like this, noting that you'll need to reference Commons.Lang in order to get the ArrayUtils.reverse functionality:

byte[] buffer = new byte[4096];
File file = new File("in.txt");
FileInputStream fileInput = new FileInputStream(file);
FileOutputStream fileOutput = new FileOutputStream("out.bin");

int index = (int)(file.length() - 4096);
int bytesRead = -1;
while((bytesRead = fileInput.read(buffer, index, 4096)) > 0 && index >= 0){
    index = Math.max(index - 4096, 0);
    if(bytesRead < 4096){
        buffer = Arrays.copyOf(buffer, bytesRead);
    }
    ArrayUtils.reverse(buffer);
    fileOutput.write(buffer);
}