Logstash 解析 json 消息

Question

我有以下日志消息需要发送到logstash

{
  "timestamp": "2017-11-02T12:25:26+0000",
  "level": "INFO",
  "thread": "dw-28 - POST /transaction",
  "mdc": {
    "requestId": "a00460f8-b27d-4a53-bafb-f9a19fa3dedb"
  },
  "logger": "app.util.JsonLogUtil",
  "message": "{\"logMessage\":\"Transaction Created\", \"transactionId\":\"612632\", \"id\":\"17\", \"customerId\":\"null\", \"reason\":\"null\", \"currentOfferedDelta\":\"null\", \"updatedAmount\":\"null\", \"majorAmount\":\"null\"}",
  "context": "default"
}

消息字段中的消息再次是 json。我想提取其所有字段并将它们添加为结果文档的一部分。 为此，我添加了如下所示的 logstash 配置文件 ->

input {
  beats {
    port => 5044

  }
}

filter {     
    json{
        source => "message"
        target => "message"
    }         
}  

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "test1"
    document_type => "%{[@metadata][type]}"
  }
}

但是弹性搜索的输出是 ->

{
  "_index": "test1",
  "_type": "log",
  "_id": "AV-QtQKVpSHmBVb3JIlV",
  "_score": 1,
  "_source": {
    "message": {
      "timestamp": "2017-11-02T12:25:26+0000",
      "level": "INFO",
      "thread": "dw-28 - POST /transaction",
      "mdc": {
        "requestId": "a00460f8-b27d-4a53-bafb-f9a19fa3dedb"
      },
      "logger": "app.util.JsonLogUtil",
      "message": "{\"logMessage\":\"Transaction Created\", \"transactionId\":\"612632\", \"id\":\"17\", \"customerId\":\"null\", \"reason\":\"null\", \"currentOfferedDelta\":\"null\", \"updatedAmount\":\"null\", \"majorAmount\":\"null\"}",
      "context": "default"
    },
    "@version": "1",
    "@timestamp": "2017-11-06T09:40:13.066Z",
    "type": "log",
    "count": 1,
    "fields": null,
    "offset": 9010,
    "input_type": "log",
    "tags": [
      "beats_input_codec_plain_applied"
    ]
  }
}

无法提取诸如 transactionId、id、reason 之类的字段 任何输入都将受到高度赞赏。

Answer 1

此过滤器可以帮助您：

filter{
mutate{
    replace => [ "message", "%{message}" ]
    gsub => [ 'message','\n','']
}

if [message] =~ /^{.*}$/{
    json { source => message }
}

}