i looking for many many time and also consult in here to found one solution to resolve problem json in elastic using logstash. my config here
input {
tcp {
port => 9000
}
}
filter{
json{
source => "message"
target => "doc"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => logstash-%{+YYYY.MM.dd}
}
}
but my elastic still have string message document not json. Document like that
{
"_index": "logstash-2017.05.12",
"_type": "logs",
"_id": "AVv8C4O4qok70-ifTOnm",
"_score": null,
"_source": {
"message": "{\"name\":\"abc\",\"id\":1494582167248}",
"@version": "1",
"@timestamp": "2017-05-12T09:42:47.263Z",
"host": "172.0.0.1",
"port": 53763
},
"fields": {
"@timestamp": [
1494582167263
]
},
"sort": [
1494582167263
]
}
Any one can help me how to fix inorder to name and id filed is as an member propeter of _source . Im expect document log like
"_source": {
"name":"abc",
"id": 1494582167248
"@version": "1",
"@timestamp": "2017-05-12T09:42:47.263Z",
"host": "192.168.2.251",
"port": 53763
}
If you want to have those fields at the root of the parsed message (which will be at the root level of _source
in ElasticSearch, you must remove the JSON target
setting. That setting specifies a parent to the parsed fields that are extracted using the JSON filter:
{
"@timestamp": "2017-05-12T11:58:40.897Z",
"port": 61981,
"@version": "1",
"host": "10.0.2.2",
"doc": {
"name": "abc",
"id": 1494582167248
},
"message": "{\"name\":\"abc\",\"id\":1494582167248}"
}
So remove the target
setting, and make sure that your index
setting is set in quotes:
input {
tcp {
port => 9000
}
}
filter{
json {
source => "message"
}
}
output {
stdout { codec => "rubydebug" }
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
Which results in:
{
"@timestamp": "2017-05-12T11:56:51.187Z",
"port": 61970,
"@version": "1",
"host": "10.0.2.2",
"name": "abc",
"id": 1494582167248,
"message": "{\"name\":\"abc\",\"id\":1494582167248}"
}
To validate with ElasticSearch curl -XGET localhost:9200/logstash-2017.05.12/_search | jq .
curl -XGET localhost:9200/logstash-2017.05.12/_search | jq .
returns
{
"_index": "logstash-2017.05.12",
"_type": "logs",
"_id": "AVv8hMTCjLo8wwWpi9R6",
"_score": 1,
"_source": {
"@timestamp": "2017-05-12T11:56:51.187Z",
"port": 61970,
"@version": "1",
"host": "10.0.2.2",
"name": "abc",
"id": 1494582167248,
"message": "{\"name\":\"abc\",\"id\":1494582167248}"
}
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.