Reading a Large File to an array java.lang.OutOfMemoryError: Java heap space

Question

I am trying to read a extremely large file to an array. it is obvious that i am not reading it to an array at the moment but that is not the problem i am having. it is about 600 million digits and this seems to cause it to get an error. Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

BufferedReader brWorld = null;
String World;
brWorld = new BufferedReader(new FileReader("Saves/" + Name));
World = brWorld.readLine();
String Numbers = World;
String[] LoadWorld = Numbers.split("");
for(int x = 0; x < MaterialArray.length * MaterialArray.length; x++){
    System.out.println(String.valueOf(LoadWorld[x]));
}

Im Loading a would into an array for my game. Is there any way of loading a text file this big? thanks Liam

Answer 1

First off, you'll need a lot of memory for this. Several GBs of RAM, not including GC room.

If you're reading a "single line" of 600M digits, in the end, after the line is actually read (you're likely not even getting this far), the string of 600M digits will require 1.2GB of memory, simply for the characters.

Java stores Strings as arrays of Characters, and Characters are stored internally as UTF-16, which is 2 bytes.

That alone sends the memory requirements through the roof.

When you do your split, you're turning each digit in to an individual string. You now have 600M Strings, and all of the over head that entails. At a minimum, you're looking at least 16 bytes per string, since it stores the pointer to the underlying array, an offset in to the array, and the length of the string. Thankfully, split will actually reuse the underlying array, but that's really not much help here.

16 bytes * 600M is 9.6GB.

Definitely heading in to the "ludicrous" realm of memory requirements now.

You don't say what you actually want to do. You want to "load the file", but you don't say in to what. So, it's hard to provide a recommendation as to how it should be done. If you want to just print the file out, then you can simply read each character one by one and print it, but that's clearly not the goal.

So, while, yes, you can "throw memory" at this problem, you don't want to. You need to study the problem and come up with a better representation of what you're trying to achieve and work from there.

Answer 2

Here is why you are running out of memory:

brWorld = new BufferedReader(new FileReader("Saves/" + Name));
World = brWorld.readLine();
String Numbers = World;
String[] LoadWorld = Numbers.split("");

You are first reading in all of the digits:

600,000,000 chars = 600,000,000 bytes = 600 MB

So already you are at 600MB at least. Then you are creating a whole new array of Strings. This means that you are creating a String pointer for each char in the array. Based on your architecture, this could be 4 bytes or more:

600 MB + 600,000,000 * 4 = 600 MB + 2,400 MB = 3,000 MB = 3 GB

*These numbers are estimates and I know the actual amount is probably MUCH larger.

As you can see, this is not a feasible solution at all. This is a MUCH better solution:

BufferedReader brWorld = null;
String world;
brWorld = new BufferedReader(new FileReader("Saves/" + Name));
world = brWorld.readLine();
for(int x = 0; x < MaterialArray.length * MaterialArray.length; x++){
    System.out.println(String.valueOf(world.charAt(x)));
}