在logstash中过滤json

Question

I have a json file with records like this one 我有一个JSON文件，其中包含这样的记录

{"id":1,"first_name":"Frank","last_name":"Mills","date":"5/31/2014","email":"fmills0@feedburner.com","country":"France","city":"La Rochelle","latitude":"46.1667","longitude":"-1.15"

and I'm trying to filter the fields in logstash, unsuccessfully so far. 到目前为止，我一直在尝试过滤Logstash中的字段，但未成功。 I tried the grok debugger and the grokconstructor but cannot make it work. 我尝试了grok 调试器和grokconstructor，但无法使其正常工作。 My last attempt is 我最后的尝试是

input {
    file{
        path => ["C:/logstash-1.4.2/mock_data.json"]
        type => "json"
        start_position => "beginning"
        sincedb_path => "/dev/null"
  }
}
filter {
  mutate {
    replace => [ "message", "%{message}" ]
  }
  json {
    source => "message"
    remove_field => "message"
  }
  mutate {
    convert => [ "latitude", "float" ]
    convert => [ "longitude","float" ]
  }
  mutate {
     rename => [ "latitude", "[location][lat]", "longitude", "[location][lon]" ]
  }
}

output {
  stdout {
    codec => rubydebug
  } 
  elasticsearch {
    host => "127.0.0.1"
    protocol => "http"
    index => "test35"
  }
}

just for the latitude and longitude but that doesn't work. 仅用于纬度和经度，但这不起作用。 Any tutorial for logstash on Json particularly. 特别是有关Json的logstash的任何教程。 Any help on this. 任何帮助。 The output for the specific configuration file is 特定配置文件的输出是

{
 "message" => "{\"id\":91,\"first_name\":\"Adam\",\"last_name\":\"Carr\",\"date\":\"11/14/2014\",\"email\":\"acarr2i@tinyurl.
com\",\"country\":\"Ghana\",\"city\":\"Mampong\",\"latitude\":\"7.06273\",\"longitude\":\"-1.4001\"},",
      "@version" => "1",
      "@timestamp" => "2015-05-04T19:05:08.409Z",
       "host" => "Toshiba",
       "path" => "C:/logstash-1.4.2/mock_data.json",
        "tags" => [
             [0] "_jsonparsefailure"
    ]
}

Updated for Alcanzar 已针对Alcanzar更新

Answer 1

The geoip filter is for adding lat/lon of an IP address to your data. geoip过滤器用于将IP地址的经/纬度添加到数据中。

Putting all of the pieces together yields this: 将所有部分放在一起将产生以下结果：

filter {
  grok {
        match => [ 'message', '(?<body>\"id\":.*\"longitude\":\"[^"]+\")' ]
        add_field => [ "json_body", "{%{body}}" ]
  }
  json {
        source => "json_body"
        remove_field => ["message","body","json_body" ]
  }
  mutate {
    convert => [ "latitude", "float" ]
    convert => [ "longitude","float" ]
  }
  mutate {
     rename => [ "latitude", "[location][lat]", 
       "longitude", "[location][lon]" ]
  }
}

Which will generate an event that looks like this: 它将生成一个如下所示的事件：

{
      "@version" => "1",
    "@timestamp" => "2015-05-04T19:48:52.051Z",
          "host" => "xxxxxxxx",
            "id" => 1,
    "first_name" => "Frank",
     "last_name" => "Mills",
          "date" => "5/31/2014",
         "email" => "fmills0@feedburner.com",
       "country" => "France",
          "city" => "La Rochelle",
      "location" => {
        "lat" => 46.1667,
        "lon" => -1.15
    }
}

which should be exactly what you want. 这应该正是您想要的。

在logstash中过滤json

问题描述

1 个解决方案

解决方案1
1 已采纳 2015-05-04 13:46:47

在logstash中过滤json

问题描述

1 个解决方案

解决方案1 1 已采纳 2015-05-04 13:46:47

解决方案1
1 已采纳 2015-05-04 13:46:47