java - 如何在时间间隔后使用log4j2或logback批量收集日志并将其写入文件

Question

I am trying to make a custom log appender using log4j2. My problem is I don't want to write log immediately to a file appender but after. So, ideally, my spring boot app should collect all the logs in some data structure and then trigger writing to file after the delay of 3 minutes in batches. (I should not use spring batch since it is not a batch application but a simple spring boot starter)

Answer 1

When I was writing my custom provider for Logback (as a part of Loki4j project) I came up with a concise implementation of a thread-safebuffer that can trigger output operation by either batch size or timeout since last output.

Usage pattern:

private static final LogRecord[] ZERO_EVENTS = new LogRecord[0];
private ConcurrentBatchBuffer<ILoggingEvent, LogRecord> buffer = new ConcurrentBatchBuffer<>(batchSize, LogRecord::create, (e, fl) -> eventFileLine(e, fl));


// somewhere in code where new log event arrives
var batch = buffer.add(event, ZERO_EVENTS);
if (batch.length > 0)
    handleBatch(batch);


// somewhere in scheduled method that triggers every timeoutMs
var batch = buffer.drain(timeoutMs, ZERO_EVENTS);
if (batch.length > 0)
    return handleBatch(batch).thenApply(r -> null);

// handling batches here
private void handleBatch(LogRecord[] lines) {
    // flush to file
}

Answer 2

Try looking at appenders that come with log4j2. Most of them implement functionality similar to what you describe. Eg RandomAccessFileAppender will write to file only after it receives complete batch of log events from async appenders infrastructure. They all share OutputStreamManager or FileManager that encapsulates this logic:

protected synchronized void write(final byte[] bytes, final int offset, final int length, final boolean immediateFlush) {
        if (immediateFlush && byteBuffer.position() == 0) {
...

Unfortunately there seems to be no time baed solution for this.

I have written log4j2 appender for Loki and it has its own ring buffer and a thread that will send a batch when there is enough data or user specified timeout passed, here :

 if (exceededBatchSizeThreshold() || exceededWaitTimeThreshold(currentTimeMillis)) {
            try {
                httpClient.log(outputBuffer);
            } finally {
                outputBuffer.clear();
                timeoutDeadline = currentTimeMillis + batchWaitMillis;
            }
        }