Parse JSON message in Logstash

Question

I am sending my jenkins logs to logstash with following config:

  redis { host => "localhost" key => "logstash" data_type => "list" codec => json } 

This works as smooth as expected, now i see follwoing message in KIBANA:

 { "_index": "logstash-2015.12.18", "_type": "logs", "_id": "AVG1BN5LXZBIbp7HE4xN", "_score": null, "_source": { "data": { "id": "965", "projectName": "NicePJ", "displayName": "#965", "fullDisplayName": "NicePJ", "url": "job/NIcePJ/965/", "buildHost": "Jenkins", "buildLabel": "master", "buildNum": 965, "buildDuration": 1, "rootProjectName": "NicePJ", "rootProjectDisplayName": "#965", "rootBuildNum": 965, "buildVariables": { "target_SUT": "0201", "report_warnings": "false", "product": "Ours", "testsuite": "Exciting_stuff5", "qft_version": "current", "target_task": "t324", "branch": "test", "testcase": "", "revision": "HEAD", "node": "hsqs960", "client": "Desktop", "run_specific_test": "false", "user": "xxxxx" } }, "message": [ "A This is a message XYZ" ], "source": "jenkins", "source_host": "http://serverXL:8080/", "@timestamp": "2015-12-18T12:16:02.000Z", "@version": 1 }, "fields": { "@timestamp": [ 1450440962000 ] }, "sort": [ 1450440962000 ] } 

Now i want to filter the message field for certain messages, but i cant get it work. How can i filter the message field and how can i access the buildHost field to use it in an if statement in the pipeline?

Following i tried after many examples:

  if[data][buildHost]== "jenkins" { grok { match => { "message[0]" => "\\[exec\\]\\s*\\<%{GREEDYDATA:test}\\s*\\[%{GREEDYDATA:result}\\]" } } } 

But this is just not working at all, please help me out.

Answer 1

Conditional

The == compares simple string and case sensitive, so "jenkins" will not match as your data shows ( "buildHost": "Jenkins", ):

if[data][buildHost]== "jenkins"

But following does:

if[data][buildHost]== "Jenkins"

If you need match both, you can either use || or regex =~ .

Grok

The grok is a filter to parse message with regex pattern. You can test your regex pattern with

• online grok debugger
• Kibana dev tools's grok debugger