在Logstash中访问嵌套的JSON字段

Question

我在logstash（最新版本）中访问嵌套的JSON字段时遇到问题。

我的配置文件如下：

input {
  http {
    port => 5001
    codec => "json"
  }
}

filter {
  mutate {
    add_field => {"es_index" => "%{[statements][authority][name]}"}
  }
  mutate {
    gsub => [
      "es_index", " ", "_"
    ]
  }
  mutate {
    lowercase => ["es_index"]
  }
  ruby {
    init => "
        def remove_dots hash
            new = Hash.new
            hash.each { |k,v|
                if v.is_a? Hash
                    v = remove_dots(v)
                end
                new[ k.gsub('.','_') ] = v
                if v.is_a? Array
                    v.each { |elem|
                        if elem.is_a? Hash
                            elem = remove_dots(elem)
                        end
                        new[ k.gsub('.','_') ] = elem
                    } unless v.nil?
                end
            } unless hash.nil?
            return new
        end
    "
    code => "
        event.instance_variable_set(:@data,remove_dots(event.to_hash))
    "
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "golab-%{+YYYY.MM.dd}"
  }  
}

我有一个带有mutate的过滤器。 我想添加一个可用作索引名称一部分的字段。 当我使用此"%{[statements][authority][name]}"时，方括号中的内容用作字符串。 %{[statements][authority][name]}保存在es_index字段中。 Logstash似乎认为这是一个字符串，但是为什么呢？

我也尝试过使用以下表达式： "%{statements}" 。 工作正常。 字段语句中的所有内容都传递给es_index 。 如果我使用"%{[statements][authority]}"则会发生奇怪的事情。 es_index填充的是"%{statements}"产生的完全相同的输出。 我想念什么？

Logstash输出为"%{[statements][authority]}" ：

{
    "statements" => {
             "verb" => {
                 "id" => "http://adlnet.gov/expapi/verbs/answered",
            "display" => {
                "en-US" => "answered"
            }
        },
          "version" => "1.0.1",
        "timestamp" => "2016-07-21T07:41:18.013880+00:00",
           "object" => {
            "definition" => {
                       "name" => {
                    "en-US" => "Example Activity"
                },
                "description" => {
                    "en-US" => "Example activity description"
                }
            },
                    "id" => "http://adlnet.gov/expapi/activities/example"
        },
            "actor" => {
               "account" => {
                "homePage" => "http://example.com",
                    "name" => "xapiguy"
            },
            "objectType" => "Agent"
        },
           "stored" => "2016-07-21T07:41:18.013880+00:00",
        "authority" => {
                  "mbox" => "mailto:info@golab.eu",
                  "name" => "GoLab",
            "objectType" => "Agent"
        },
               "id" => "0771b9bc-b1b8-4cb7-898e-93e8e5a9c550"
    },
            "id" => "a7e31874-780e-438a-874c-964373d219af",
      "@version" => "1",
    "@timestamp" => "2016-07-21T07:41:19.061Z",
          "host" => "172.23.0.3",
       "headers" => {
              "request_method" => "POST",
                "request_path" => "/",
                 "request_uri" => "/",
                "http_version" => "HTTP/1.1",
                   "http_host" => "logstasher:5001",
              "content_length" => "709",
        "http_accept_encoding" => "gzip, deflate",
                 "http_accept" => "*/*",
             "http_user_agent" => "python-requests/2.9.1",
             "http_connection" => "close",
                "content_type" => "application/json"
    },
      "es_index" => "{\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/answered\",\"display\":{\"en-us\":\"answered\"}},\"version\":\"1.0.1\",\"timestamp\":\"2016-07-21t07:41:18.013880+00:00\",\"object\":{\"definition\":{\"name\":{\"en-us\":\"example_activity\"},\"description\":{\"en-us\":\"example_activity_description\"}},\"id\":\"http://adlnet.gov/expapi/activities/example\",\"objecttype\":\"activity\"},\"actor\":{\"account\":{\"homepage\":\"http://example.com\",\"name\":\"xapiguy\"},\"objecttype\":\"agent\"},\"stored\":\"2016-07-21t07:41:18.013880+00:00\",\"authority\":{\"mbox\":\"mailto:info@golab.eu\",\"name\":\"golab\",\"objecttype\":\"agent\"},\"id\":\"0771b9bc-b1b8-4cb7-898e-93e8e5a9c550\"}"
}

您可以看到授权是es_index一部分。 因此，它没有被选为领域。

提前谢谢了

Answer 1

我找到了解决方案。 学分归jpcarey（ Elasticsearch论坛 ）

我必须删除codec => "json" 。 这导致了另一个数据结构。 statements现在是数组，而不是对象。 所以我需要将%{[statements][authority][name]}更改为%{[statements][0][authority][name]} 。 那没有问题。

如果您遵循给定的链接，您还会发现我的mutate过滤器更好的实现。