为 elasticsearch 中归档的消息创建自定义 grok 模式

Question

I am having a query related to grok processor. For example this is my message filed

{
"message":"agentId:agent003"
}

I want to Grok this and my output should me something like this

{
"message":"agentId:agent003",
"agentId":"agent003"
}

Could some one help me on this how to achieve this? If i am able to do it for one field i can manage for rest of my fields. Thanks in advance.

This is the pipeline i have created in elasticsearch

PUT _ingest/pipeline/dissectpipeline
{
  "description" : "split message content",
  "processors": [
    {
      "dissect": {
      "field": "message",
      "pattern" : "%{apm_application_message.agentId}:%{apm_application_message.agentId}"
   }
    }
  ]
}

Central management added filebeat module other config

- pipeline:
    if: ctx.first_char == '{'
    name: '{< IngestPipeline "dissectpipeline" >}'

There is no error with my filebeat it's working fine but i am unable to find any field like apm_application_message.agentId in index.

How to make sure my pipeline working or not. Also if i am doing something wrong please let me know.

Answer 1

Instead of grok I'd suggest using the dissect filter instead with, which is more intuitive and easier to use.

dissect {
  mapping => {
    "message" => "%{?agentId}:%{&agentId}"
  }
}

If you're using Filebeat, there is also the possibility to use the dissect processor :

processors:
  - dissect:
      tokenizer: "%{?agentId}:%{&agentId}"
      field: "message"
      target_prefix: ""