[英]how to store my json log file to logstash with json filter
This is my json log file. 这是我的json日志文件。 I'm trying to store the file to my elastic-Search through my logstash.
我正在尝试通过logstash将文件存储到我的Elastic-Search中。
{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre":
["Action", "Adventure", "Sci-Fi"] }
after storing the data into the elasticSearch, my results is as follow 将数据存储到elasticSearch中后,我的结果如下
{
"_index": "filebeat-6.2.4-2018.11.09",
"_type": "doc",
"_id": "n-J39mYB6zb53NvEugMO",
"_score": 1,
"_source": {
"@timestamp": "2018-11-09T03:15:32.262Z",
"source": "/Users/jinwoopark/Jin/json_files/testJson.log",
"offset": 106,
"message": """{ "id": "135569", "title" : "Star Trek Beyond", "year":2016 , "genre":["Action", "Adventure", "Sci-Fi"] }""",
"id": "%{id}",
"@version": "1",
"host": "Jinui-MacBook-Pro.local",
"tags": [
"beats_input_codec_plain_applied"
],
"prospector": {
"type": "log"
},
"title": "%{title}",
"beat": {
"name": "Jinui-MacBook-Pro.local",
"hostname": "Jinui-MacBook-Pro.local",
"version": "6.2.4"
}
}
}
What I'm trying to do is that, 我想做的是
I want to store only "genre value" into the message field, and store other values(ex id, title) into extra fields(the created fields, which is id and title field). 我只想将“类型值”存储到消息字段中,并将其他值(例如id,title)存储到额外的字段(创建的字段,即id和title字段)中。 but the extra fields were stored with empty values(%{id}, %{title}).
但多余的字段存储为空值(%{id},%{title})。 It seems like I need to modify my logstash json filter, but here I need your help.
似乎我需要修改logstash json过滤器,但是在这里我需要您的帮助。
my current configuration of logstash is as follow 我当前的logstash配置如下
input {
beats {
port => 5044
}
}
filter {
json {
source => "genre" //want to store only genre (from json log) into message field
}
mutate {
add_field => {
"id" => "%{id}" // want to create extra field for id value from log file
"title" => "%{title}" // want to create extra field for title value from log file
}
}
date {
match => [ "timestamp", "dd/MM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
When you tell the json filter that the source
is genre
, it should ignore the rest of the document, which would explain why you don't get an id
or title
. 当您告诉json过滤器
source
是genre
,它应该忽略文档的其余部分,这将解释为什么没有id
或title
。
Seems like you should parse the entire json document, and use the mutate->replace plugin to move the contents of genre
to message
. 似乎您应该解析整个json文档,并使用mutate-> replace插件将
genre
的内容移动到message
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.