Logstash grok pattern for pgpool logs

Question

I have a problem finding a right grok pattern for all my logs in order to parse all of them through logstash.

Example of my logs:

1) 2016-04-13 19:55:40: pid 21950: LOG: pool_send_and_wait: Error or notice message from backend: : DB node id: 4 backend pid: 65156 statement: "UPDATE certname......"... (The rest of the log doesn't matter)

2) 2016-04-13 19:55:40: pid 17555: FATAL: failed to read kind from backend

My grok pattern (which works partially) :

grok { match => { "message" => "%{GREEDYDATA:logdate}: pid %{NUMBER:pid}: %{LOGLEVEL:loglevel}: %{GREEDYDATA:logmessage}" } }

As you can see these are the things I care about: 1) The date of the log 2) The PID 3) The loglevel 4) The message itself

Except from the pattern above I also tried to specify the date using the \\A%{TIMESTAMP_ISO8601:timestamp} pattern (just like the grok debugger suggested me).

My assumption is that the first log is the kind of my problematic logs because it got the word "log" two times in it, Although in the second time it doesn't appears after a colon (unlike my grok pattern) so I have no idea why it fails the parsing operation.

In fact, my logs which doesn't contains the word "log" twice parsed successfully - my pattern works partially.

Thanks :)

Answer 1

The trick to making grok patterns is to start at the left, get the first one working, and move slowly to the right until you're done.

Doing that, you would see that it stops working when you add %{LOGLEVEL}. Why? Because "LOG" is not a valid valid for LOGLEVEL. It's expecting things like DEBUG, etc. Use something like %{WORD} instead and it works fine.