Java多次打开和关闭要写入的文件

Question

Below is a class I have which writes a ConcurrentMap<String, List<String>> to a file. The key in the map is the path, and the value in the map are to be written sequentially to the file. This Task<Void> gets called every time there are 1,000 values in the map:

public class MapWriter extends Task<Void> {

private final ParsingProducerConsumerContext context;

public MapWriter(ParsingProducerConsumerContext context) {
    this.context = context;
}

@Override
protected Void call() throws Exception {
    if (!isCancelled() || !context.isEmpty()) {
        ConcurrentMap<String, List<String>> jsonObjectMap = context.fetchAndReset();

        jsonObjectMap.entrySet().forEach((t) -> {                
            try {
                FileUtils.writeLines(new File(context.getPath() + t.getKey() + "\\sorted.json"), t.getValue(), true);
            } catch (IOException ex) {
                context.getLogger().log("Error writing to disk:");
                context.getLogger().log(ex.toString());
                context.stopEverything();
            }
        });

        context.getLogger().log(jsonObjectMap.values().stream().mapToInt(List::size).sum() + " schedules written to disk ");
    } else {
        context.getLogger().log("Nothing to write");
    }

    return null;
}
}

All the while this task is running, there is a producer Task reading a ~2GByte file line by line, which gets processed by a consumer and placed into ConcurrentMap<String, List<String>> .

Whilst this does work, it is very slow!

My research suggests that there is a significant enough overhead in opening and closing files repeatedly to impair performance, so was wondering if the following approach might be better?

Maintain a Map<String, File> of File objects which are open. If the key in ConcurrentMap<String, List<String>> corresponds to an open file, use that File reference for writing When all processing has finished, loop over Map<String, File> values and close each file.

Does this sound a sensible way to go? There would be approx 100 files open though.

EDIT :: I did a simple benchmark using System.nanoTime() . The file being imported line by line by the producer is approx 2GB, and each line is between 6kb and 10kb (in the List<String> ).

Also, an OutOfMemory error is encountered! I guess because the 2GByte is effectively loaded into memory, and not being written out quickly enough?

514 jsonObjects written to disk in 2258007ms 538 jsonObjects written to disk in 2525166ms 1372 jsonObjects written to disk in 169959ms 1690 jsonObjects written to disk in 720824ms 9079 jsonObjects written to disk in 5221168ms 22552 jsonObjects written to disk in 6943207ms 13392 jsonObjects written to disk in 6475639ms 0 jsonObjects written to disk in 6ms 0 jsonObjects written to disk in 5ms 0 jsonObjects written to disk in 5ms 40 jsonObjects written to disk in 23108ms 631 jsonObjects written to disk in 200269ms 3883 jsonObjects written to disk in 2054177ms Producer failed with java.lang.OutOfMemoryError: GC overhead limit exceeded

For completeness, here is the Producer class:

public class NRODJsonProducer extends Task<Void> {

private final ParsingProducerConsumerContext context;

public NRODJsonProducer(ParsingProducerConsumerContext context) {
    this.context = context;
}

@Override
protected Void call() throws Exception {
    context.getLogger().log("Producer created");

    LineIterator li = FileUtils.lineIterator(new File(context.getPath() + context.getFilterFile()));

    while (li.hasNext()) {
        try {
            context.getQueue().put(li.next());
        } catch (InterruptedException ex) {
            Logger.getLogger(NRODJsonProducer.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    LineIterator.closeQuietly(li);

    context.getLogger().log("Producer finished...");

    return null;
}

}

Answer 1

I don't see why. This code writes out everything for a key to a file with the same name, then moves on to the next key. If the producer produces another entry for that key, it overwrites the previous entry, and this code will write the file again. Keeping files open won't help that.

The real problem seems to be that you keep writing the same data to the file, because you never remove a processed key from the map.

NB Your util condition is wrong. It should be

if (!isCancelled() && !context.isEmpty())