Grok patterns format in logstash

Question

Mostly my patterns working but after Framework Error in :] part does not see in the logs.

How can print rest of it on logs. I tried at the last in patterns, but still does not work %{GREEDYDATA:restofthem}

My grok Pattern:

\[%{TIMESTAMP_ISO8601:ServerTimestamp}\|%{WORD:Log4netHostname}\|%{DATA:ProjectName}\|%{DATA:TestName}\|%{DATA:UserName}\|%{DATA:ClientIP}\|%{DATA:ClientMachineName}\|%{LOGLEVEL:LogLevel}\|%{DATA:method}\|%{DATA:message}\|%{GREEDYDATA:Exception}|%{GREEDYDATA:Exception}\]%{GREEDYDATA:restofthem}

My Log:

[2018-06-05 13:26:57,641|host1|Appname|TTA|KKM|112.310.104.722|Host23|ERROR|Logger.Log4Net|LogError|Framework Error in :]
WebDriverTimeoutException: Timed out after 5 seconds
 at DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)
 at DefaultWait`1.Until[TResult](Func`2 condition)

Answer 1

The reason is, there is a new line after :] and You need to match \\n character in order to parse it. There are various different ways to do it, you can match a new line using \\n character like this,

\[%{TIMESTAMP_ISO8601:ServerTimestamp}\|%{WORD:Log4netHostname}\|%{DATA:ProjectName}\|%{DATA:TestName}\|%{DATA:UserName}\|%{DATA:ClientIP}\|%{DATA:ClientMachineName}\|%{LOGLEVEL:LogLevel}\|%{DATA:method}\|%{DATA:message}\|%{GREEDYDATA:Exception}\n%{GREEDYDATA:2ndLine}\n%{GREEDYDATA:3rdLine}\n%{GREEDYDATA:4thLine}

which will produce,

  ......

  "Exception": [
    [
      "Framework Error in :]"
    ]
  ],
  "2ndLine": [
    [
      "WebDriverTimeoutException: Timed out after 5 seconds"
    ]
  ],
  "3rdLine": [
    [
      " at DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)"
    ]
  ],
  "4thLine": [
    [
      " at DefaultWait`1.Until[TResult](Func`2 condition)"
    ]
  ]

OR,

use ?m to match all lines after the first line in one block like this,

\[%{TIMESTAMP_ISO8601:ServerTimestamp}\|%{WORD:Log4netHostname}\|%{DATA:ProjectName}\|%{DATA:TestName}\|%{DATA:UserName}\|%{DATA:ClientIP}\|%{DATA:ClientMachineName}\|%{LOGLEVEL:LogLevel}\|%{DATA:method}\|%{DATA:message}\|%{GREEDYDATA:Exception}(?m)%{GREEDYDATA:everythingelse}

Output,

  .....
  ],
  "Exception": [
    [
      "Framework Error in :]"
    ]
  ],
  "everythingelse": [
    [
      "\nWebDriverTimeoutException: Timed out after 5 seconds\n at DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)\n at DefaultWait`1.Until[TResult](Func`2 condition)"
    ]
  ]
}