Java Buffered reader running out of heap space

Question

I'm trying to parse a very large file (~1.2 GB). Some lines of the file are bigger than the maximum allowed String size.

        FileReader fileReader = new FileReader(filePath);
        BufferedReader bufferedReader = new BufferedReader(fileReader);

        while ((line = bufferedReader.readLine()) != null) {
              //Do something
        }

        bufferedReader.close();

Error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
    at java.lang.StringBuffer.append(StringBuffer.java:369)
    at java.io.BufferedReader.readLine(BufferedReader.java:370)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at sax.parser.PrettyPrintXML.format(PrettyPrintXML.java:30)

line 30 :

while ((line = bufferedReader.readLine()) != null) {

Can anyone suggest any alternative approach for this case.

Answer 1

You are using readLine() on a file that doesn't have lines. So it tries to read the entire file as a single lines. This does not scale.

Solution: don't. Read a chunk at a time, or maybe even a character at a time: whatever is dictated by the unstated structure of your file.

Answer 2

I believe maximum string character length is 2^31-1 [2,147,483,647] and 1.2GB txt file(assuming is a txt file) can store about 1,200,000,000 characters. Why do you need to read all the data? What are you using it for? Can you split the file up into several files or read and parse it as a smaller string. Need more info.

Answer 3

You can use Apache commons IO :
https://commons.apache.org/proper/commons-io/description.html

example:

 InputStream in = new URL( "http://commons.apache.org" ).openStream();
 try {
   System.out.println( IOUtils.toString( in ) );
 } finally {
   IOUtils.closeQuietly(in);
 }