如何使用json過濾器將我的json日志文件存儲到logstash

Question

這是我的json日志文件。 我正在嘗試通過logstash將文件存儲到我的Elastic-Search中。

{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre": 
["Action", "Adventure", "Sci-Fi"] }

將數據存儲到elasticSearch中后，我的結果如下

{
    "_index": "filebeat-6.2.4-2018.11.09",
    "_type": "doc",
    "_id": "n-J39mYB6zb53NvEugMO",
    "_score": 1,
    "_source": {
      "@timestamp": "2018-11-09T03:15:32.262Z",
      "source": "/Users/jinwoopark/Jin/json_files/testJson.log",
      "offset": 106,
      "message": """{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre":["Action", "Adventure", "Sci-Fi"] }""",
      "id": "%{id}",
      "@version": "1",
      "host": "Jinui-MacBook-Pro.local",
      "tags": [
        "beats_input_codec_plain_applied"
      ],
      "prospector": {
        "type": "log"
      },
      "title": "%{title}",
      "beat": {
        "name": "Jinui-MacBook-Pro.local",
        "hostname": "Jinui-MacBook-Pro.local",
        "version": "6.2.4"
      }
    }
  }

我想做的是

我只想將“類型值”存儲到消息字段中，並將其他值（例如id，title）存儲到額外的字段（創建的字段，即id和title字段）中。 但多余的字段存儲為空值（％{id}，％{title}）。 似乎我需要修改logstash json過濾器，但是在這里我需要您的幫助。

我當前的logstash配置如下

input {
    beats {
     port => 5044
    }
}

filter {
    json {
            source => "genre" //want to store only genre (from json log) into message field 
    }
    mutate {
            add_field => {
                    "id" => "%{id}" // want to create extra field for id value from log file
                    "title" => "%{title}" // want to create extra field for title value from log file
            }
    }
    date {
            match => [ "timestamp", "dd/MM/yyyy:HH:mm:ss Z" ]
    }
}
output {
    elasticsearch {
            hosts => ["http://localhost:9200"]
            index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
    stdout {
            codec => rubydebug
    }
}

Answer 1

當您告訴json過濾器source是genre ，它應該忽略文檔的其余部分，這將解釋為什么沒有id或title 。

似乎您應該解析整個json文檔，並使用mutate-> replace插件將genre的內容移動到message 。