Splunk regex line breaker, combining multiple ERROR [stderr] lines into one event

Question

I am working on displaying logs from docker json-file in splunk. For the most part I have it working, except for when a stack trace is printed. I know e.printStackTrace() isn't best practice, but our services have a few here and there that I want to support in Splunk.

I am configuring the props.conf file, and I have the following LINE_BREAKER regex in props.conf. LINE_BREAKER=([\\n\\r]+)\\s*{"log":"[0-9]+.*[0-9]\\s+

This will match up to the end of a timestamp of this form: {"log":"2019-04-18 15:18:18,796 ERROR [stderr] ...

The problem is that the printed stack trace is coming in multiple lines like this:

{"log":"2019-04-18 15:18:18,796 ERROR [stderr] (default-threads - 3) java.util.NoSuchElementException: No value present\n","stream":"stdout","time":"2019-04-18T19:18:18.800485539Z"}
{"log":"2019-04-18 15:18:18,804 ERROR [stderr] (default-threads - 3) at java.util.Optional.get(Optional.java:135)\n","stream":"stdout","time":"2019-04-18T19:18:18.806510971Z"}
{"log":"2019-04-18 15:18:19,259 ERROR [stderr] (default-threads - 3) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n","stream":"stdout","time":"2019-04-18T19:18:19.264689098Z"}
{"log":"2019-04-18 15:18:19,259 ERROR [stderr] (default-threads - 3) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n","stream":"stdout","time":"2019-04-18T19:18:19.281810119Z"}
{"log":"2019-04-18 15:18:19,274 ERROR [stderr] (default-threads - 3) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n","stream":"stdout","time":"2019-04-18T19:18:19.28185714Z"}
{"log":"2019-04-18 15:18:19,275 ERROR [stderr] (default-threads - 3) at java.lang.reflect.Method.invoke(Method.java:498)\n","stream":"stdout","time":"2019-04-18T19:18:19.281867696Z"}
{"log":"2019-04-18 15:18:19,275 ERROR [stderr] (default-threads - 3) at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49)\n","stream":"stdout","time":"2019-04-18T19:18:19.281875844Z"}
{"log":"2019-04-18 15:18:19,498 ERROR [org.apache.activemq.ActiveMQSession] (default-threads - 3) error dispatching message: : javax.ejb.EJBTransactionRolledbackException\n","stream":"stdout","time":"2019-04-18T19:18:19.523359025Z"}

... and so on. until the end of the stack trace. Each line ends up being an event based on my regex for LINE_BREAKER , which is fine for INFO messages and single line ERROR messages from our logger, but not for the stack trace as above.

I want to combine this stack trace of the ERROR [stderr] together into one event. So my line break should match until the next timestamp message that is not [stderr].

from my example, it should match from the first line with ERROR [stderr] until: {"log":"2019-04-18 15:18:19,498 (last log line above).

I have tried adding negative lookahead but now it will only match 1 time. ([\\n\\r]+)\\s*{"log":"[0-9]+.*[0-9]\\s+[\\s\\S]+\\[(?!stderr]) matches until the last line, but if there are lines under that, none of them match anymore.

Answer 1

In this part of the pattern {"log":"[0-9]+.*[0-9]\\s+ you are matching the timestamp part using [0-9]+.*[0-9]\\s+ can cause unnecessary bracktracking due to the .* which will first match until the end of the string.

You could make the timestamp like pattern more specific instead like [0-9]{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d+

To get the matches, one option could be to match the line containing stderr . Then repeat matching the following lines with that pattern 0+ times starting with a newline to combine them.

To keep matching the other lines you could make use of a negative lookahead after ERROR and an alternation .

^(?:{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR \[stderr\].*(?:\n{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR \[stderr\].*)*|{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR (?!\[stderr\]).*)

• ^ Start of the string
• (?: Non capturing group
  • {"log":"[0-9]{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d+ ERROR \\[stderr\\].*
  • (?: Non capturing group
    • \\n{"log":"[0-9]{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d+ ERROR \\[stderr\\].*
  • )* Close non capturing group and repeat 0+ times
  • | Or
  • {"log":"[0-9]{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d+ ERROR (?!\\[stderr\\]).*
• ) Close non capturing group

Regex demo