how to store my json log file to logstash with json filter

Question

This is my json log file. I'm trying to store the file to my elastic-Search through my logstash.

{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre": 
["Action", "Adventure", "Sci-Fi"] }

after storing the data into the elasticSearch, my results is as follow

{
    "_index": "filebeat-6.2.4-2018.11.09",
    "_type": "doc",
    "_id": "n-J39mYB6zb53NvEugMO",
    "_score": 1,
    "_source": {
      "@timestamp": "2018-11-09T03:15:32.262Z",
      "source": "/Users/jinwoopark/Jin/json_files/testJson.log",
      "offset": 106,
      "message": """{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre":["Action", "Adventure", "Sci-Fi"] }""",
      "id": "%{id}",
      "@version": "1",
      "host": "Jinui-MacBook-Pro.local",
      "tags": [
        "beats_input_codec_plain_applied"
      ],
      "prospector": {
        "type": "log"
      },
      "title": "%{title}",
      "beat": {
        "name": "Jinui-MacBook-Pro.local",
        "hostname": "Jinui-MacBook-Pro.local",
        "version": "6.2.4"
      }
    }
  }

What I'm trying to do is that,

I want to store only "genre value" into the message field, and store other values(ex id, title) into extra fields(the created fields, which is id and title field). but the extra fields were stored with empty values(%{id}, %{title}). It seems like I need to modify my logstash json filter, but here I need your help.

my current configuration of logstash is as follow

input {
    beats {
     port => 5044
    }
}

filter {
    json {
            source => "genre" //want to store only genre (from json log) into message field 
    }
    mutate {
            add_field => {
                    "id" => "%{id}" // want to create extra field for id value from log file
                    "title" => "%{title}" // want to create extra field for title value from log file
            }
    }
    date {
            match => [ "timestamp", "dd/MM/yyyy:HH:mm:ss Z" ]
    }
}
output {
    elasticsearch {
            hosts => ["http://localhost:9200"]
            index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
    stdout {
            codec => rubydebug
    }
}

Answer 1

When you tell the json filter that the source is genre , it should ignore the rest of the document, which would explain why you don't get an id or title .

Seems like you should parse the entire json document, and use the mutate->replace plugin to move the contents of genre to message .