Log4J2 JsonLayout printing out class, method and line and @timestamp override

Question

So we have an app that we are going to send all of its logs to Kibana server (through Kafka server).

Follows is the basics we managed to get working, posting without issues to Kibana:

        <Kafka name="KafkaAppender" topic="topic1">
        <JsonLayout compact="true">
            <KeyValuePair key="service" value="some_app_tag"/>
            <KeyValuePair key="@timestamp" value="${date:yyyy-MM-dd HH:mm:ss.SSS}"/>
            <KeyValuePair key="host_name" value="${hostName}"/>
            <KeyValuePair key="unique_id" value="$${map:name:-NA}"/>
    </JsonLayout>
        <Property name="bootstrap.servers">kafka1.com:9092,kafka2.com:9092,kafka3.com:9092</Property>
    </Kafka>

However we are finding it difficult to get class, method and line number to print out in the JsonLayout so that it could be indexed by elasticSearch and hence searchable in Kibana as a field.

We tried various syntax combinations/variations for those 3 fields, such as %c{2} , %M and %L for Line - they all print out literally as the variables we try to put in.

In the Console Appender, they work using: <pattern>%23.23d{yyyy-MM-dd HH:mm:ss.SSS} %5p [%t] [$${map:name:-NA}] %c{1}.%M(%F:%L): %highlight{%m%n%throwable}</pattern>

"$${map:name:-NA}" is something we added to add unique error ID that can be searched based on error message the user recieves, it is being set by something like:

    StringMapMessage mapMsg = new StringMapMessage();
    mapMsg.put("name", "arun");
    LOGGER.fatal(mapMsg);

And this works, our only issues are

1. Outputting class, line and method in the JSON for Kafka input

2. We were so far not able to override Kibana's @timestamp field, so that it contains the timestamp generated by log4j, so that we can reliably sort by time/date the logs were generated (otherwise they would be confusing if they came in random order) - when we add the @ sign, Kibana adds a tag like _timestampparsefailure , and displays the submitted @timestamp with _@timestamp , instead of using it to override the timestamp it generates.

Can anyone advice please? We have searched far and wide, and so far did not find anything close.

Thanks.

Answer 1

We have not found a way to make JsonLayout output class, line and method, the workaround we employed is by implementing our own layout, basically creating a new custom layout plugin as that is basically a duplicate of JsonLayout class, with added support for Patterns:

@Plugin(name = "CustomJsonLayout", category = Node.CATEGORY, elementType = Layout.ELEMENT_TYPE, printObject = true)
public final class CustomJsonLayout extends AbstractJacksonLayout {

Which has added parameters in its constructor:

final boolean stacktraceAsString,
final boolean includeNullDelimiter,
final KeyValuePair[] additionalFields,
final boolean objectMessageAsJsonObject

Then of course added them to the super call as well.

The biggest change is by overriding the toSerializable(final LogEvent event, final Writer writer) method and adding private method customFunctionToApplyPatternConversion which looks something along the lines of:

Object wrappedEvent = wrapLogEvent(convertMutableToLog4jEvent(event));
if (wrappedEvent instanceof LogEventWithAdditionalFields) {
    LogEventWithAdditionalFields eventWithAdditionalFields = (LogEventWithAdditionalFields) wrappedEvent;
    eventWithAdditionalFields = customFunctionToApplyPatternConversion(event, eventWithAdditionalFields);

    wrappedEvent = eventWithAdditionalFields;
}
objectWriter.writeValue(writer, wrappedEvent);

Most of the stuff is done inside the customFunctionToApplyPatternConversion method, but I cant post that as I am not authorized to.

This is the general idea of how to add support for detecting patterns and replacing them with values. You can get a better grasp of this if you look at the complete code for JsonLayout classes, and PatternLayout classes - this is essentially merging them together, then specifying CustomJsonLayout in the log4j2.xml instead of the default one.

Answer 2

在Log4j2配置文件的JSON属性中添加“ locationInfo =“ true”“：例如：

<JsonLayout complete="false" locationInfo="true" properties="true" propertiesAsList="true" eventEol="true">