如何在logstash的grok模式中使用IF ELSE条件

Question

I have web and API log combined and I want to save it separately in elasticsearch. So I want to write one pattern if the request is for API then if past should execute, the request is web then else part of the log should be executed.

Below are few web and API logs.

00:06:27,778 INFO  [stdout] (ajp--0.0.0.0-8009-38) 00:06:27.777 [ajp--0.0.0.0-8009-38] INFO  c.r.s.web.rest.WidgetController - Method getWidgetDetails() started to get widget details.
00:06:27,783 INFO  [stdout] (ajp--0.0.0.0-8009-38) ---> HTTP GET http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true
00:06:27,817 INFO  [stdout] (ajp--0.0.0.0-8009-38) <--- HTTP 200 http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true (29ms)
00:06:27,822 INFO  [stdout] (ajp--0.0.0.0-8009-38) 00:06:27.822 [ajp--0.0.0.0-8009-38] INFO  c.r.s.web.rest.WidgetController - Method getWidgetDetails() finished.
00:06:27,899 INFO  [stdout] (ajp--0.0.0.0-8009-40) 00:06:27.899 [ajp--0.0.0.0-8009-40] INFO  c.r.s.web.controller.LoginController - Inside initLoginPage() of LoginController

I tried to write condition but it's not working. It's working only up to thread name. After thread I have multiple type log so not able to write witout if condition.

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)

Can anybody give me suggestion?

Answer 1

You don't need to use an if/else conditon to do this, you can use multiple patterns, one will match the API log lines and the other will match the WEB log lines.

For the API log lines you can use the following pattern:

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\\[%{NOTSPACE}\\]%{SPACE}\\(%{NOTSPACE:THREAD}\\)%{SPACE}(?:%{DATA})%{SPACE}\\[%{DATA}\\]%{SPACE}%{WORD}%{SPACE}%{GREEDYDATA:MSG}

And your return will be something like this:

{
"MSG": "c.r.s.web.controller.LoginController - Inside initLoginPage() of LoginController",
"CREATED_ON": "00:06:27,899",
"LEVEL": "INFO",
"THREAD": "ajp--0.0.0.0-8009-40"
}

For the web lines you can use the following pattern:

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\\[%{NOTSPACE}\\]%{SPACE}\\(%{NOTSPACE:THREAD}\\)%{SPACE}%{DATA}%{WORD:PROTOCOL}%{SPACE}%{WORD:MethodOrStatus}%{SPACE}%{GREEDYDATA:ENDPOINT}

And the result will be:

{
"CREATED_ON": "00:06:27,783",
"PROTOCOL": "HTTP",
"ENDPOINT": "http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true",
"LEVEL": "INFO",
"THREAD": "ajp--0.0.0.0-8009-38",
"MethodOrStatus": "GET"
}

To use multiple patterns in grok just do this:

grok {
  match => ["message", "pattern1", "pattern2"]
}

Or you can save your patterns to a file and use patterns_dir to point to the directory of the file.

If you still want to use a conditional, just check for anything in the message, for example:

if "HTTP" in [message] { 
 grok { your grok for the web messages }
} else {
 grok { your grok for the api messages }
}