在Java中使用扫描仪时如何减少内存泄漏？

Question

I using Scanner for reading many logs file (~100 files), each logs file is ~120mb and have over 1,000,000 lines. The memory using for reading there logs file is continuous increase and cause memory overload. How to prevents this happen? Here's my code:

File file = processing.poll(); // processing is a queue.
Scanner sc = new Scanner(file);
String line;
int lineCount = 0;
while (sc.hasNextLine()) {
    line = sc.nextLine();
}
sc.close();

Thanks you!!

P/S: The memory increase slow, ~1mb for each file processed but I will push this code to the server and there're countless files in the future.

Answer 1

I suggest you use a try-with-resources to close the Scanner and release the file handle. Also, you could restrict the scope of the line and I think you meant to increment the lineCount . Something like,

int lineCount = 0;
try (Scanner sc = new Scanner(file)) {
    while (sc.hasNextLine()) {
        String line = sc.nextLine();
        lineCount++;
    }
}

Alternatively, you might use a finally block like

int lineCount = 0;
Scanner sc = new Scanner(file);
try {
    while (sc.hasNextLine()) {
        String line = sc.nextLine();
        lineCount++;
    }
} finally {
    sc.close();
}

Answer 2

The problem is that you are loading all of these huge ~100 log files into memory. Think about it. ~100 log files, that are ~120mb each. That's a lot of memory being taken! You are going to have to read in a single log file, process it, then release the handle on the file and remove it from memory, and then repeat until all the logs have been processed how you want them to. You need an algorithm that will process them separately and release the handle/memory as needed to avoid the overload.