读取一行内容非常长的文件时，发生Java内存不足错误

Question

Our application need to read a file with a single line and that single line contains large amount data . What we are doing is that , read the line from file and store it in string and tokenize the string with - and store to a list . From that list some entries are to be checked.

the method is as follows

public bollean checkMessage(String filename){
boolean retBool = true;
LinkedList tokenList;
int size;
String line = "";
try {
    File file = new File(filename);
    FileInputStream fs = new FileInputStream(file);
    InputStreamReader is = new InputStreamReader(fs);
    BufferedReader br = new BufferedReader(is);
    while ((line = br.readLine()) != null) {
        line.trim();
        tokenList = tokenizeString(line, "-");
        if (tokenList == null) {
            retBool = false;
            resultMsg = "Error in  File.java "                  
        }
        if (retBool) {
                retBool = checkMessagePart(tokenList);
        }
   }
}

the error occurs at line , while ((line = br.readLine()) != null)

error is

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2367)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
    at java.lang.StringBuffer.append(StringBuffer.java:322)
    at java.io.BufferedReader.readLine(BufferedReader.java:363)
    at java.io.BufferedReader.readLine(BufferedReader.java:382)

Actually increasing heapsize didn't work. the size of the file trying to read is more than 1gb. Also tried to read as chunks of bytes , but when adding the read data to StringBuilder or list will again generate the MemoryError

Answer 1

If the problem is that you cannot read the file to a String, then don't do it. Read it token by token by using some other method. The easy one is using Scanner with the right delimiter ("-" in your case). If you find its performance lacking, you could resort to implementing your own version of BufferedReader in which the "lines" are split by that character instead of the normal values.