java.lang.OutOfMemoryError：读取大文本文件时超出了GC开销限制

Question

I have a text file which is about 2 GB in size. Each line of the file has the following format:

some text possibly separated by commas , unique integer

I need to take each line, split it into two parts : text, unique integer and put it in a Hashmap as a key value pair.

Now I am facing OutOfMemory Error even when the heap size is set to 10 GB.

There could be two reasons for this : 1. The way I am reading the file is wrong. 2. I am creating too many unnecessary String objects.

This is what I am doing :

InputStream is = Thread.currentThread().getContextClassLoader().getResourceAsStream("filename.txt");

InputStreamReader stream = new InputStreamReader(is, StandardCharsets.UTF_8);

BufferedReader reader = new BufferedReader(stream);

while(true)
{
 line =reader.readLine();
 if (line == null) {
  break;
 }
 String text= line.substring(0, line.lastIndexOf(",")).trim();

 String id = line.substring(line.lastIndexOf(",") + 1).trim();

 //put this in a hashmap and other processing
}

Since I need to split each line of the text in two parts and the first part(text) might have commas as well, I am using substring() method for this purpose.

The reason I am using trim is that I need to put the text and id in the Hashmap without trailing and leading whitespaces.

Error message:

 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.Arrays.copyOfRange(Arrays.java:3664)
    at java.lang.String.<init>(String.java:207)
    at java.lang.String.substring(String.java:1969)

Answer 1

u should add loop condition. please try it again with the underline code.it seems to work!

    try {
        String line;

        while ((line = reader.readLine()) != null) {
            String text = line.substring(0, line.lastIndexOf(",")).trim();

            String id = line.substring(line.lastIndexOf(",") + 1).trim();

            //put this in a hashmap and other processing
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            reader.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }