grok pattern to parse the logs in logstash

Question

I'm trying to parse various logs from grok, but it's too difficult, so I'm asking for your help.

I want to analyze url and mapping information in the log, but it doesn't work well because of other logs.

I tried this way but I don't think it was properly parsed.

"\[%{TIMESTAMP_ISO8601:logtime}\]\[%{DATA:thead_id}\]\[%{DATA:level}\]\[%{DATA:method}\]%{SPACE}%{WORD:http}/%{WORD:https}%{SPACE}%{WORD:request}%{SPACE}:%{SPACE}%{WORD:aop}.RequestInfoModel\(url\=%{WORD:url}%{GREEDYDATA:param}"

please help.

log example:

[21-03-17 08:23:57][scheduled-task-1][DEBUG][c.f.a.b.CleanExpiredAccessInfo.cleanExpiredAccessInfo()] [batch] Clear expired accessInfoStore : store_size=0 (CleanExpiredAccessInfo.java:23)

[21-03-16 00:11:49][http-nio-0.0.0.0-8080-exec-10][INFO ][c.f.a.c.a.GuardianAOP.onAroundHandler()] HTTP/HTTPS Request : GuardianAOP.RequestInfoModel(url=/api/v3/me/extra/key, mapping=GET, parameter=AccessToken(userKey=test@mail.com, clientKey=da3ac9db59dfa32002e7fae6849d06, userType=User, accessibleClient={1daec78593214e6b53ce9803ded5916=Manager, da3ac9db59dfa32002e7fae6849d06=User})) (GuardianAOP.java:57)

[21-03-11 16:54:11][http-nio-18080-exec-8][ERROR][c.f.c.biz.v3.UserBiz.getValidMemberInfoVo()] BizException : [rt_code:9999] not found user (UserBiz.java:228)

Answer 1

[21-03-16 00:11:49][http-nio-0.0.0.0-8080-exec-10][INFO ][c.f.a.c.a.GuardianAOP.onAroundHandler()] HTTP/HTTPS Request : GuardianAOP.RequestInfoModel(url=/api/v3/me/extra/key, mapping=GET, parameter=AccessToken(userKey=test@mail.com, clientKey=da3ac9db59dfa32002e7fae6849d06, userType=User, accessibleClient={1daec78593214e6b53ce9803ded5916=Manager, da3ac9db59dfa32002e7fae6849d06=User})) (GuardianAOP.java:57)

For the above mentioned log,

\[%{TIMESTAMP_ISO8601:logtime}\]\[%{DATA:thread_id}\]\[%{WORD:level}( )\]\[%{DATA:method}\]%{SPACE}%{WORD:HTTP}/%{WORD:HTTPS}%{SPACE}%{WORD:Request}%{SPACE}:%{SPACE}%{WORD:aop}.RequestInfoModel\(%{GREEDYDATA:api_info}\)%{SPACE}\(%{DATA:file_name}:%{NUMBER:line_number}\)

This works perfectly. If you want to slpit the api_info into more parts, that's upto you and as per your requirements.

Please use this for more debugging.

Now, the other two log samples that you have provided are different. Either you have to write an extremely generic pattern, or you can write multiple patterns and match with either.

Have a look at the accepted answer on this question . It explains it quite neatly.