Passing dynamic value to log4j2 xml config

Question

I am trying to write a logging module using log4j2 that needs to write the logs to console STDOUT in the form of json.

For that am trying to use PatternLayout in the form of JSON as below.

I have some difficulties in passing values from my code dynamically to log4j2.xml config file to substitute them on run time while writing the log.

I tried using StructuredDataMessage and MapMessages to substitute the values from Map as mentioned in the https://logging.apache.org/log4j/2.0/manual/lookups.html .

I also tried StrLookup and ContextMaplookup as well but no sucess so far.

Below is my xml config

    <?xml version="1.0" encoding="UTF-8"?>
    <Configuration status="WARN" name="App" packages="com.test.common.logging">
        <Properties>
            <Property name="filename">target/rolling1/rollingtest.log</Property>
            <Property name="maptype">$${map:type}</Property>

        </Properties>

        <ThresholdFilter level="debug"/>

        <Appenders>
            <Console name="Console" target="SYSTEM_OUT">
              <PatternLayout pattern="%highlight{{'logged time': '%d{dd MMM yyyy HH:mm:ss}', 
            'LEVEL' : '%level',
            'CLASS' : '%c{-1}',
            'Module' : '[%t]',
            'message' : '%m',
            'error' : '%exception',
            'class' : '%C',
            'threadid' : '%tid',
            'threadname' : '%thread',
            'whatisthis' : '${filename}',
            'processid' : '%pid', 
            'logdir' : '$${sd:type}'
            'location' : '${log4j:configLocation}'
            'systemproperty' : '$${ctx:key-}'
            }}%n"/>
            </Console>
        </Appenders>

        <Loggers>
            <Root level="trace">
                <AppenderRef ref="Console"/>
            </Root>
        </Loggers>

    </Configuration>

Below is my code where am trying to pass dynamic values using Strucutured Data message, strlookup and mapmessage

public class App 
{


    public static void main( String[] args )
    {
        Logger logger = LogManager.getLogger(App.class);

//      ConfigurationBuilder<BuiltConfiguration> builder
//       = ConfigurationBuilderFactory.newConfigurationBuilder();
//      
//      LayoutComponentBuilder standard 
//        = builder.newLayout("PatternLayout");
//      standard.
//      
        System.out.println( "Hello World!" );
        StructuredDataMessage message = new StructuredDataMessage("1", "name", "string");
        message.put("1", "nme");
       // MapMessage mapm = new MapMessage(map)
        MapMessage map = new MapMessage();
        map.put("type", "value");
        map.put("key", "value");
        map.put("name", "arun");
        StrLookup strlook = new StrLookup() {

            public String lookup(LogEvent event, String key) {
                // TODO Auto-generated method stub
                return null;
            }

            public String lookup(String key) {
                // TODO Auto-generated method stub

                return "value";
            }
        };
        ContextMapLookup lookup = new ContextMapLookup();
        System.out.println(lookup.lookup("key"));
        System.out.println(strlook.lookup("key"));

       // MapLookup.setMainArguments(args);

        System.setProperty("log_dir", App.class.getSimpleName());;
        logger.trace("trace log message");
        logger.debug("Debug log message");
        logger.info("Info log message");
        logger.error("Error log message");
        logger.fatal("Fatal log message");
        logger.info("Info log message[]");
        logger.error("null exception[]", new NullPointerException());

        // Lay

    }
}

My Output:

Hello World!
null
value
[36m{'logged time': '24 Aug 2018 12:32:51',    'LEVEL' : 'DEBUG',   'CLASS' : 'test.common.logging.App',   'Module' : '[main]',   'message' : 'Debug log message',   'error' : '',   'class' : 'com.test.common.logging.App',   'threadid' : '1',   'threadname' : 'main',   'whatisthis' : 'target/rolling1/rollingtest.log',   'processid' : 'DEBUGid',    'logdir' : '${sd:type}'   'location' : '/Users/parunkarthick/ferry-commons/common/logging/target/classes/log4j2.xml'   'systemproperty' : '${ctx:key}'   [m}
[32m{'logged time': '24 Aug 2018 12:32:51',    'LEVEL' : 'INFO',   'CLASS' : 'test.common.logging.App',   'Module' : '[main]',   'message' : 'Info log message',   'error' : '',   'class' : 'com.test.common.logging.App',   'threadid' : '1',   'threadname' : 'main',   'whatisthis' : 'target/rolling1/rollingtest.log',   'processid' : 'INFOid',    'logdir' : '${sd:type}'   'location' : '/Users/parunkarthick/ferry-commons/common/logging/target/classes/log4j2.xml'   'systemproperty' : '${ctx:key}'   [m}
[1;31m{'logged time': '24 Aug 2018 12:32:51',    'LEVEL' : 'ERROR',   'CLASS' : 'test.common.logging.App',   'Module' : '[main]',   'message' : 'Error log message',   'error' : '',   'class' : 'com.test.common.logging.App',   'threadid' : '1',   'threadname' : 'main',   'whatisthis' : 'target/rolling1/rollingtest.log',   'processid' : 'ERRORid',    'logdir' : '${sd:type}'   'location' : '/Users/parunkarthick/ferry-commons/common/logging/target/classes/log4j2.xml'   'systemproperty' : '${ctx:key}'   [m}
[1;31m{'logged time': '24 Aug 2018 12:32:51',    'LEVEL' : 'FATAL',   'CLASS' : 'test.common.logging.App',   'Module' : '[main]',   'message' : 'Fatal log message',   'error' : '',   'class' : 'com.test.common.logging.App',   'threadid' : '1',   'threadname' : 'main',   'whatisthis' : 'target/rolling1/rollingtest.log',   'processid' : 'FATALid',    'logdir' : '${sd:type}'   'location' : '/Users/parunkarthick/ferry-commons/common/logging/target/classes/log4j2.xml'   'systemproperty' : '${ctx:key}'   [m}
[32m{'logged time': '24 Aug 2018 12:32:51',    'LEVEL' : 'INFO',   'CLASS' : 'test.common.logging.App',   'Module' : '[main]',   'message' : 'Info log message[]',   'error' : '',   'class' : 'com.test.common.logging.App',   'threadid' : '1',   'threadname' : 'main',   'whatisthis' : 'target/rolling1/rollingtest.log',   'processid' : 'INFOid',    'logdir' : '${sd:type}'   'location' : '/Users/parunkarthick/ferry-commons/common/logging/target/classes/log4j2.xml'   'systemproperty' : '${ctx:key}'   [m}
[1;31m{'logged time': '24 Aug 2018 12:32:51',    'LEVEL' : 'ERROR',   'CLASS' : 'test.common.logging.App',   'Module' : '[main]',   'message' : 'null exception[]',   'error' : ' java.lang.NullPointerException
    at com.test.common.logging.App.main(App.java:69)
',   'class' : 'com.test.common.logging.App',   'threadid' : '1',   'threadname' : 'main',   'whatisthis' : 'target/rolling1/rollingtest.log',   'processid' : 'ERRORid',    'logdir' : '${sd:type}'   'location' : '/Users/parunkarthick/ferry-commons/common/logging/target/classes/log4j2.xml'   'systemproperty' : '${ctx:key}'   [m}

If you see the lastvalue systemproperty is not reflecting as expected by substituting from the code value.

Answer 1

I think you have some fundamental misunderstandings about log4j2. Rather than me trying to list all of the issues I see in your code, I think the best thing to do is for me to provide some sample code and explain the output. I think when you see some working code you'll understand where you went wrong.

For the purposes of this example I simplified your log4j2 configuration file by removing the elements that appear to be working and focusing on those that are not. I changed the PatternLayout to the following:

<PatternLayout pattern="{'LEVEL' : '%level', 'typeFromStructMsg' : '${sd:type}', 'contextValue' : '${ctx:myContextKey}', 'nameFromMapMsg' : '${map:name}', 'mySysProperty' : '${sys:mySysProperty}'}%n" />

I also modified the App class you provided:

package example;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
import org.apache.logging.log4j.message.MapMessage;
import org.apache.logging.log4j.message.StringFormattedMessage;
import org.apache.logging.log4j.message.StringMapMessage;
import org.apache.logging.log4j.message.StructuredDataMessage;


public class App {

    private static final Logger logger = LogManager.getLogger();

    public static void main( String[] args )
    {
        ThreadContext.put("myContextKey", "myContextValue");

        StructuredDataMessage structMsg = new StructuredDataMessage("1", "name", "string");

        StringMapMessage mapMsg = new StringMapMessage();
        mapMsg.put("name", "arun");

        System.setProperty("mySysProperty", "sys prop value");

        logger.info(mapMsg);
        logger.warn(structMsg);

        logger.error("Error log message");
    }
}

When the App class is run, the following console output is generated:

{'LEVEL' : 'INFO', 'typeFromStructMsg' : '${sd:type}', 'contextValue' : 'myContextValue', 'nameFromMapMsg' : 'arun', 'mySysProperty' : 'sys prop value'}
{'LEVEL' : 'WARN', 'typeFromStructMsg' : 'string', 'contextValue' : 'myContextValue', 'nameFromMapMsg' : '${map:name}', 'mySysProperty' : 'sys prop value'}
{'LEVEL' : 'ERROR', 'typeFromStructMsg' : '${sd:type}', 'contextValue' : 'myContextValue', 'nameFromMapMsg' : '${map:name}', 'mySysProperty' : 'sys prop value'}

Notice that in the first line of output we see this: 'nameFromMapMsg': 'arun' while in other lines we see this: 'nameFromMapMsg': '${map:name}'

The first line of output is generated by this line of code: logger.info(mapMsg); which passes an instance of a MapMessage called mapMsg to the info method. Since the message is an instance of MapMessage and it contains a key called name the map lookup will replace ${map:name} with the value it finds for the name key within the message. This is why only the first line of output shows 'nameFromMapMsg': 'arun' - the other output lines are generated from messages that are not an instance of MapMessage .

Similarly, notice how in the second line of output we see 'typeFromStructMsg': 'string' . This is because the log was generated from a StructuredDataMessage , which was defined with a type of "string":

StructuredDataMessage structMsg = new StructuredDataMessage("1", "name", "string");

In other lines of the output we did not pass a StructuredDataMessage so in those lines we see 'typeFromStructMsg': '${sd:type}' as log4j2 can't find a value for the type.

Finally, notice how that in all output lines we see this: 'mySysProperty': 'sys prop value' . This is because the System Properties Lookup does not depend on the type of message passed to the logger. This lookup is always able to find a value for the system property mySysProperty because we defined it:

System.setProperty("mySysProperty", "sys prop value");

and, as I said before, system properties are independent of the message (they are not stored within the message).

The same is true for 'contextValue': 'myContextValue' - the ThreadContext is independent of the message and since we defined a value for this key:

ThreadContext.put("myContextKey", "myContextValue");

the lookup is always able to find the value regardless of the kind of message sent to the logger.

I hope this example code helps to illustrate how some of the lookups are used and some of how the architecture of log4j2 is designed. Best of luck!

Answer 2

It is not feasible to edit the PatternLayout in the xml file itself, if the values are static and will remain the same then it works, else you need to implement custom pattern by extending inbuilt methods and then format your message accordingly. The following link will let you know the same.

https://blog.10pines.com/2020/03/02/log-custom-json-with-log4j2/