filter json in logstash
Question
I have a json file with records like this one
{"id":1,"first_name":"Frank","last_name":"Mills","date":"5/31/2014","email":"fmills0@feedburner.com","country":"France","city":"La Rochelle","latitude":"46.1667","longitude":"-1.15"
and I'm trying to filter the fields in logstash, unsuccessfully so far. I tried the grok debugger and the grokconstructor but cannot make it work. My last attempt is
input {
file{
path => ["C:/logstash-1.4.2/mock_data.json"]
type => "json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
mutate {
replace => [ "message", "%{message}" ]
}
json {
source => "message"
remove_field => "message"
}
mutate {
convert => [ "latitude", "float" ]
convert => [ "longitude","float" ]
}
mutate {
rename => [ "latitude", "[location][lat]", "longitude", "[location][lon]" ]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
host => "127.0.0.1"
protocol => "http"
index => "test35"
}
}
just for the latitude and longitude but that doesn't work. Any tutorial for logstash on Json particularly. Any help on this. The output for the specific configuration file is
{
"message" => "{\"id\":91,\"first_name\":\"Adam\",\"last_name\":\"Carr\",\"date\":\"11/14/2014\",\"email\":\"acarr2i@tinyurl.
com\",\"country\":\"Ghana\",\"city\":\"Mampong\",\"latitude\":\"7.06273\",\"longitude\":\"-1.4001\"},",
"@version" => "1",
"@timestamp" => "2015-05-04T19:05:08.409Z",
"host" => "Toshiba",
"path" => "C:/logstash-1.4.2/mock_data.json",
"tags" => [
[0] "_jsonparsefailure"
]
}
Updated for Alcanzar
1 answers
solution1
1 ACCPTED 2015-05-04 13:46:47
The geoip filter is for adding lat/lon of an IP address to your data.
Putting all of the pieces together yields this:
filter {
grok {
match => [ 'message', '(?<body>\"id\":.*\"longitude\":\"[^"]+\")' ]
add_field => [ "json_body", "{%{body}}" ]
}
json {
source => "json_body"
remove_field => ["message","body","json_body" ]
}
mutate {
convert => [ "latitude", "float" ]
convert => [ "longitude","float" ]
}
mutate {
rename => [ "latitude", "[location][lat]",
"longitude", "[location][lon]" ]
}
}
Which will generate an event that looks like this:
{
"@version" => "1",
"@timestamp" => "2015-05-04T19:48:52.051Z",
"host" => "xxxxxxxx",
"id" => 1,
"first_name" => "Frank",
"last_name" => "Mills",
"date" => "5/31/2014",
"email" => "fmills0@feedburner.com",
"country" => "France",
"city" => "La Rochelle",
"location" => {
"lat" => 46.1667,
"lon" => -1.15
}
}
which should be exactly what you want.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.