Cannot handle multiline input from filebeats in logstash with grok

Question

I am new with logstash and I have the following multiline input in logstash from filebeats:

"message":"[step info]\\nstep: 3\\ngrammar name: grammar1\\nnoInputTimeout: 6000\\nstep stream idle time: 14910\\nstep stream start time: 2017-12-01 17:06:10.024\\nrec start time: 2017-12-01 17:06:09.994\\nrec finish time: 2017-12-01 17:06:12.748\\nsystem prompt duration: 570\\nuser barged in: true\\nuser noInput time: 0\\nuser speech duration: 1190\\nspeech start trigger: 8265\\nspeech start on rec: 7945\\nspeech end trigger: 8415\\nspeech end on rec: 9135\\nrec completion cause: 000 success\\nrec completion type: SR\\nrec result: onetwothreefour\\nrec inputMode: speech\\nRTF: 0.47\\nrec process time: 557\\nrec latency: 61\\nrec post delay: 62"

I am trying to find a pattern with grok but nothing is working. I also tried it with split and mutate but cannot manage to make it work.

Answer 1

This is just a pointer for you to try but the KV filter , (key value filter) may help.

Looking at your example you could do something like this.

kv {
  source => "message"
  field_split => "\n"
  value_split => ":"
}

This would take your example of

"message":"[step info]\nstep: 3\ngrammar name: grammar1\nnoInputTimeout: 6000\nstep stream idle time: 14910\nstep stream start time: 2017-12-01 17:06:10.024\nrec start time: 2017-12-01 17:06:09.994\nrec finish time: 2017-12-01 17:06:12.748\nsystem prompt duration: 570\nuser barged in: true\nuser noInput time: 0\nuser speech duration: 1190\nspeech start trigger: 8265\nspeech start on rec: 7945\nspeech end trigger: 8415\nspeech end on rec: 9135\nrec completion cause: 000 success\nrec completion type: SR\nrec result: onetwothreefour\nrec inputMode: speech\nRTF: 0.47\nrec process time: 557\nrec latency: 61\nrec post delay: 62"

And split on the \\n char, then create key value pairs out of the examples you have, the left hand side would be the field name and the right hand side would be the value.

step: 3
grammar name: grammar1
noInputTimeout: 6000
step stream idle time: 14910
step stream start time: 2017-12-01 17:06:10.024
rec start time: 2017-12-01 17:06:09.994
rec finish time: 2017-12-01 17:06:12.748

If you need to split out the [step info] then you would need to provide more examples, but I would be using greedydata to basically split you're content into 2 fields, 1 for [step info] and 1 for [step lines] and pass split lines as the source field for KV filter above.

Hope this points you in the right direction.

E.