logstash / grok自定义文件

Question

I'm a totally new user of ELK stack. I've got a little problem with filtering out specific section from my log.

Sample log:

[2017-05-30 13:58:09,336] INFO  [com.qwerty.test.core.services.impl.order.OrderEntryService] (OrderEntryService.java:5426) [http-/0.0.0.0:1111-111] {{CT,1496145487308}{IP,111.11.111.11}{JTX,1511059/176275501}{OBJT,goodsMovement.reportsUtils.ConsignmentStocksList}{OPT,SQ}{PID,111111}{SS,SSCPLTMPRODPL}{TRT,SAP_LOGISTIC_REPORT}{UID,StudentSaSo-8}}: Saving order: K1010101

and my grok filter:

grok {
    match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM2}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},%{NONNEGINT})\] %{LOGLEVEL:loglevel} * \[(?<logger>[A-Za-z0-9$_.]+)\] \(%{JAVAFILE:class}:%{NONNEGINT:line}\) \[%{NOTSPACE:thread}\] %{GREEDYDATA:message_TEST}"}
    }

so i need filter out "ID", "PID", IP and "UID" and i have no idea how to configure this specific custom pattern. I try use patterns from https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns but it does not work for me

Answer 1

The beginning of your grok is fine. Instead of the last part %{GREEDYDATA:message_TEST} , which will just save the remainder of your message into message_TEST you should actually perform parsing of your object. Something like this will save PID , IP and UID fields from your object into respective variables (keep in mind that this pattern relies on the fields order):

{{.*}{IP,%{IPV4:IP}}({.*}){3}{PID,%{POSINT:PID}}({.*}){2}{UID,%{DATA:UID}}}

Now a little explanation of what it does. Outer pair of curly brackets is the limit of your object. Then we will take care of each field limited by pair of curly brackets inside of the object.

• first: opening curly bracket { ;
• then, the first inner field is {CT,1496145487308} , we are not interested in saving it - so just tell grok that there's some string limited by curly brackets: {.*} ;
• next goes field with IP, that we need to save: {IP,111.11.111.11} . It starts with curly bracket and IP, followed by IP address that we have to save (last IP in the match is the name of variable that will store the IP address): {IP,%{IPV4:IP}
• now we have three groups of strings surrounded by curly brackets that we don't need to save: {JTX,1511059/176275501}{OBJT,goodsMovement.reportsUtils.ConsignmentStocksList}{OPT,SQ} . For grok it looks like: ({.*}){3} ;
• then goes PID field: {PID,111111} . For grok PID is just a positive integer similarly to IP surrounded by curly brackets and with PID, in front: {PID,%{POSINT:PID}} ;
• two more groups that we don't want to save. Skip them similarly to the previous ones: ({.*}){2} ;
• last field is: {UID,StudentSaSo-8} , which is just a string of data for grok. Similarly to IP and PID saving it in the respective variable: {UID,%{DATA:UID}} ;
• finally we have closing curly bracket: } .

In the end your final grok will look as follows:

(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},%{NONNEGINT})\] %{LOGLEVEL:loglevel} * \[(?<logger>[A-Za-z0-9$_.]+)\] \(%{JAVAFILE:class}:%{NONNEGINT:line}\) \[%{NOTSPACE:thread}\] {%{DATA}{IP,%{IPV4:IP}}({.*}){3}{PID,%{POSINT:PID}}({.*}){2}{UID,%{DATA:UID}}}