使用Readread读取Java中的大文件

Question

I have a big file of size 10gb, If i read its whole contents using readfully() in java, I get a outofmemoryerror, so i decided to read the big 10gb file in parts using same readfully(), for this i need to pass the offset and length parameters for readfully(). The offset must be of long or double datatype so that it can point to different parts of the file. But the readfully() accepts only int offset. How to read the big data?

try {
    IOUtils.readFully(in, contents, minOffset, maxOffset);
    value.set(contents, 0, contents.length);
} finally {
    IOUtils.closeStream(in);
}

Can I use seek() to get to a specific position and then use readfully() from that position?

Answer 1

Use the class java.util.Scanner to run through the contents of the file and retrieve lines serially, one by one:

FileInputStream inputStream = null;
Scanner sc = null;
try {
    inputStream = new FileInputStream(path);
    sc = new Scanner(inputStream, "UTF-8");
    while (sc.hasNextLine()) {
        String line = sc.nextLine();
        // System.out.println(line);
    }
    // note that Scanner suppresses exceptions
    if (sc.ioException() != null) {
        throw sc.ioException();
    }
}
finally {
    if (inputStream != null) {
        inputStream.close();
    }
    if (sc != null) {
        sc.close();
    }
}

This solution will iterate through all the lines in the file – allowing for processing of each line – without keeping references to them – and in conclusion, without keeping them in memory. For more details see this .