Regexp for parse log with fluentd

Question

I'm trying to parse application log with some regexp. I was able to parse the timestamp. But after that, If I try to add more expressions to the fluentd format the first attribute "time" disappears with giving me an exception. And no records are matched.

I'm using Fluentular.

Log message will be something like:

Date=[2018-04-11 08:44:30,219] Thread=[20] Level=[INFO] EventId=[2] Message=[Request finished in 1.1825ms 200 text/plain; charset=utf-8]

This is the first regexp with the first key (There is an issue with milliseconds but is not important). In the attributes I can see the key Time with this value.

First regexp ok

then If I try to isolate more info on the log I lose the first key and I'm unable to parse any other data.

Regexp with second key

What I'm doing wrong?

Thanks

UPDATE:

based on comments now I have this expression

(?<time>\[(?<time>[^\]\[]+)])\s+(?<Thread>\S+)\s+(?<Level>\S+)\s+(?<EventId>\S+)\s+(?<Message>[^ ].*$)

Regexp 3

Is almost perfect. The only problem is that, the expression also get the square bracket

Key Thread Value Thread=[20]

I don't know if this expression can be improved to avoid the

"Thread=[]"

UPDATE 2:

In order to do test, I installed fluentd via apt with the plugin and do some tests and my final setup is something like

<parse>
        @type kv
        time_key Date
        types Date:time:%Y-%m-%d %H:%M:%S,Thread:integer,Level:string,EventId:integer,Message:string
        kv_delimiter /\]\s+/
        kv_char "=["

Seems better now:

2018-04-11 08:44:30.219000000 +0200 kv_log: {"Thread":20,"Level":"INFO","EventId":2,"Message":"Request finished in 1.1825ms 200 text/plain; charset=utf-8]"}

Answer 1

I'd suggest to parse the key-value pairs with the Key-Value Pairs Parser Plugin for Fluentd .

Parameters

• kv_delimiter /\\]\\s+(?=\\w+=)/ (or just /\\]\\s+/ ) (see how the kvp pairs are split here )
• kv_char "=[" (the string will be used to split the key from the value)