Read a huge file of numbers in Java in a memory-efficient way?

Question

In Java , having a file of 335Gb size that contains individual numbers at each line, I need to read it line by line like if it was a stream of numbers - I must not keep all the data in memory. I was told that Scanner class will not work. Could you please recommend the best possible way to do that?

Answer 1

None of the java.io input stream classes would "keep all the data in memory". I think you are free to choose what is best for you such as BufferedReader or DataInputStream etc.

Answer 2

If you use BufferedReader you should be able to get up to 90 MB/s in one thread.

You can use trick to break up the file and read portion of the data concurrently, but this will only help if your disk read through put is high.

For example you can memory map 335 GB into memory at once without using the heap much. This will work even if you have a fraction of this amount of main memory.

What is the read transfer rate you can get with your disk subsystem?