NGINX自定义日志的弹性搜索索引模板

Question

我有NGINX的以下日志：

111.111.111.111、11.11.11.11-11.11.11.11 [06 / May / 2016：08：26：10 +0000]“ POST / some-service / GetSomething HTTP / 1.1” 499 0“-”“ Jakarta Commons-HttpClient / 3.1 “” 7979798797979799“ 59.370- ” {\\ x0A \\ x22correlationId \\ x22：\\ x22TestCorr1 \\ x22 \\ x0A}

Logstash将如下所示：

input { stdin {} }
output { stdout { codec => "rubydebug" } }

filter {
   grok {
 match => { "message" => "%{COMBINEDAPACHELOG} %{QS:partner_id} %{NUMBER:req_time} %{GREEDYDATA:extra_fields}" }
 add_field => [ "received_at", "%{@timestamp}" ]
 add_field => [ "received_from", "%{host}" ]
  }


mutate {
 gsub => ["extra_fields", "\"","",
      "extra_fields", "\\x0A","",
      "extra_fields", "\\x22",'\"',
      "extra_fields", "(\\)",""
  ]
}

json {
  source => "extra_fields"
  target => "extra_fields_json"
}


mutate {
  add_field => {
    "correlationId" => "%{[extra_fields_json][correlationId]}"
  }
}

}

问题是req_time是字符串，因此我需要使用以下模板转换为float：

{
  "template" : "filebeat*",
 "settings" : {
   "index.refresh_interval" : "5s"
 },

 "mappings" : {
  "properties" : {
    "@timestamp": { "type": "date" },
    "partner_id": { "type": "string", "index": "not_analyzed" },
    "@version": { "type": "string", "index": "not_analyzed" },
    "req_time" : { "type" : "float", "index" : "not_analyzed" }, 
    "res_time" : { "type" : "string", "index" : "not_analyzed" },
    "purchaseTime" : { "type" : "date", "index" : "not_analyzed" },
    "received_at" : { "type" : "date", "index" : "not_analyzed" },
    "itemPrice" : { "type" : "double", "index" : "not_analyzed" },
    "total" : { "type" : "integer", "index" : "not_analyzed" },
    "bytes" : { "type" : "double", "index" : "not_analyzed" }
  }
   }
 }

}

验证使用：curl -XGET'http ：// localhost：9200 / filebeat-2016.06.30 / _mapping / field / req_time '

我正进入（状态：

{“ filebeat-2016.06.30”：{“ mappings”：{“ nginxlog”：{“ req_time”：{“ full_name”：“ req_time”，“ mapping”：{ “ req_time”：{“ type”：“ string” } }}}}}}

所以我的模板肯定不起作用。 有人可以帮忙吗？

Answer 1

最后，我只是删除了模板，然后让ES来宾访问字段类型。 它确实起作用。