读取大文件错误“ outofmemoryerror”（java）

Question

sorry for my english. I want to read a large file, but when I read error occurs outOfMemoryError . I do not understand how to work with memory in the application. The following code does not work:

try {

    StringBuilder fileData = new StringBuilder(1000);
    BufferedReader reader = new BufferedReader(new FileReader(file));

    char[] buf = new char[8192];
    int bytesread = 0, 
        bytesBuffered = 0;

    while( (bytesread = reader.read( buf )) > -1 ) {

        String readData = String.valueOf(buf, 0, bytesread);
        bytesBuffered += bytesread;

        fileData.append(readData); //this is error

        if (bytesBuffered > 1024 * 1024) { 
            bytesBuffered = 0;
        }
    }

    System.out.println(fileData.toString().toCharArray());
} finally {

}

Answer 1

Try this. This might be helpful :-

try{
    BufferedReader reader = new BufferedReader(new FileReader(file));
    String txt = "";
    while( (txt = reader.read()) != null){
        System.out.println(txt);
    }
}catch(Exception e){
   System.out.println("Error : "+e.getMessage());
}

Answer 2

You need pre allocate a large buffer to avoid reallocate.

File file = ...;
StringBuilder fileData = new StringBuilder(file.size());

And running with large heap size:

java -Xmx2G

==== update

A while loop using buffer doesn't need too memory to run. Treat input like a stream, match your search string with the stream. It's a really simple state machine. If you need search multiple words, you can find a TrieTree implementation(support stream) for that.

// the match state model
...xxxxxxabxxxxxaxxxxxabcdexxxx...
         ab     a     abcd

    File file = new File("path_to_your_file");
    String yourSearchWord = "abcd";
    int matchIndex = 0;
    boolean matchPrefix = false;
    try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
        int chr;
        while ((chr = reader.read()) != -1) {
            if (matchPrefix == false) {
                char searchChar = yourSearchWord.charAt(0);
                if (chr == searchChar) {
                    matchPrefix = true;
                    matchIndex = 0;
                }
            } else {
                char searchChar = yourSearchWord.charAt(++matchIndex);
                if (chr == searchChar) {
                    if (matchIndex == yourSearchWord.length() - 1) {
                        // match!!
                        System.out.println("match: " + matchIndex);
                        matchPrefix = false;
                        matchIndex = 0;
                    }
                } else {
                    matchPrefix = false;
                    matchIndex = 0;
                }
            }
        }
    }

Answer 3

You should not hold such big files in memory, because you run out of it, as you see. Since you use Java 7, you need to read the file manually as stream and check the content on the fly. Otherwise you could use the stream API of Java 8. This is just an example. It works, but keep in mind, that the position of the found word could vary due to encoding issues, so this is no production code:

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;

public class FileReader
{
    private static String wordToFind = "SEARCHED_WORD";
    private static File file = new File("YOUR_FILE");
    private static int currentMatchingPosition;
    private static int foundAtPosition = -1;
    private static int charsRead;

    public static void main(String[] args) throws IOException
    {
        try (FileInputStream fis = new FileInputStream(file))
        {
            System.out.println("Total size to read (in bytes) : " + fis.available());

            int c;
            while ((c = fis.read()) != -1)
            {
                charsRead++;
                checkContent(c);
            }

            if (foundAtPosition > -1)
            {
                System.out.println("Found word at position: " + (foundAtPosition - wordToFind.length()));
            }
            else
            {
                System.out.println("Didnt't find the word!");
            }

        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }

    private static void checkContent(int c)
    {
        if (currentMatchingPosition >= wordToFind.length())
        {
            //already found....
            return;
        }

        if (wordToFind.charAt(currentMatchingPosition) == (char)c)
        {
            foundAtPosition = charsRead;
            currentMatchingPosition++;
        }
        else
        {
            currentMatchingPosition = 0;
            foundAtPosition = -1;
        }
    }
}