解析CSV时出现OutOfMemoryError

Question

I have one huge csv file (500MB) and 400k records in it

id, name, comment, text
1, Alex, Hello, I believe in you

Column text consist much information and sentences. I want to get this column("Text"), replace all non-alphabetic symbols to " " and sort it in reverse order with from most frequent words in column "Text" to most infrequent with limit 1000. This is how it looks like. I'm using CsvReader library

CsvReader doc = new CsvReader("My CSV Name");
        doc.readHeaders();
        try {
            List<String> listWords = new ArrayList<>();
            while (doc.readRecord()) {
                listWords.addAll(Arrays.asList(doc.get("Text"/*my column name*/).replaceAll("\\P{Alpha}", " ").toLowerCase().trim().split("[ ]+")));
            }

            Map<String, Long> sortedText = listWords.stream()
                    .collect(groupingBy(chr -> chr, counting()))
                    .entrySet().stream()
                    .sorted(Map.Entry.comparingByValue(Collections.reverseOrder()))
                    .limit(1000)
                    .collect(Collectors.toMap(
                            Map.Entry::getKey,
                            Map.Entry::getValue,
                            (e1, e2) -> e1,
                            LinkedHashMap::new
                    ));
            sortedText.forEach((k, v) -> System.out.println("Word: " + k + " || " + "Count: " + v));
            doc.close();
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            doc.close();
        }

After running I have out of memory error that my GC exceeded. How to do it best? I can't increase my heap size, I just need to work with default settings

Answer 1

A suggestion for the problem: Instead of adding all words in the listWords , try to make the accounting of words by each CSV line processed.

The code would be something like this:

CsvReader doc = null;

try {

    doc = new CsvReader(""My CSV Name");
    doc.readHeaders();

    Map<String, Long> mostFrequent = new HashMap<String, Long>();

    while (doc.readRecord()) {

        Arrays.asList(doc.get("text"/*my column name*/).replaceAll("\\P{Alpha}", " ").toLowerCase().trim().split("[ ]+")).
        stream().forEach(word -> {

            if (mostFrequent.containsKey(word)) {
                mostFrequent.put(word, mostFrequent.get(word) + 1);  
            }
            else {
                mostFrequent.put(word, 1l);
            }
        });
    }

    Map<String, Long> sortedText = mostFrequent.entrySet().stream()
        .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
        .limit(1000)
        .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,
                (e1, e2) -> e1, LinkedHashMap::new));

    sortedText.forEach((k, v) -> System.out.println("Word: " + k + " || " + "Count: " + v));

    doc.close();

} catch (IOException e) {
    e.printStackTrace();
} finally {
    doc.close();
}