NGINX自定义日志的弹性搜索索引模板

Question

I have the following log from NGINX:

111.111.111.111, 11.11.11.11 - 11.11.11.11 [06/May/2016:08:26:10 +0000] "POST /some-service/GetSomething HTTP/1.1" 499 0 "-" "Jakarta Commons-HttpClient/3.1" "7979798797979799" 59.370 - "{\\x0A\\x22correlationId\\x22 : \\x22TestCorr1\\x22\\x0A}"

Logstash will be like this:

input { stdin {} }
output { stdout { codec => "rubydebug" } }

filter {
   grok {
 match => { "message" => "%{COMBINEDAPACHELOG} %{QS:partner_id} %{NUMBER:req_time} %{GREEDYDATA:extra_fields}" }
 add_field => [ "received_at", "%{@timestamp}" ]
 add_field => [ "received_from", "%{host}" ]
  }


mutate {
 gsub => ["extra_fields", "\"","",
      "extra_fields", "\\x0A","",
      "extra_fields", "\\x22",'\"',
      "extra_fields", "(\\)",""
  ]
}

json {
  source => "extra_fields"
  target => "extra_fields_json"
}


mutate {
  add_field => {
    "correlationId" => "%{[extra_fields_json][correlationId]}"
  }
}

}

The problem is req_time is string, so I need to convert to float using the following template:

{
  "template" : "filebeat*",
 "settings" : {
   "index.refresh_interval" : "5s"
 },

 "mappings" : {
  "properties" : {
    "@timestamp": { "type": "date" },
    "partner_id": { "type": "string", "index": "not_analyzed" },
    "@version": { "type": "string", "index": "not_analyzed" },
    "req_time" : { "type" : "float", "index" : "not_analyzed" }, 
    "res_time" : { "type" : "string", "index" : "not_analyzed" },
    "purchaseTime" : { "type" : "date", "index" : "not_analyzed" },
    "received_at" : { "type" : "date", "index" : "not_analyzed" },
    "itemPrice" : { "type" : "double", "index" : "not_analyzed" },
    "total" : { "type" : "integer", "index" : "not_analyzed" },
    "bytes" : { "type" : "double", "index" : "not_analyzed" }
  }
   }
 }

}

Verified using: curl -XGET ' http://localhost:9200/filebeat-2016.06.30/_mapping/field/req_time '

I am getting:

{"filebeat-2016.06.30":{"mappings":{"nginxlog":{"req_time": {"full_name":"req_time","mapping":{ "req_time":{"type":"string"} }}}}}}

so my template definitely does not work. Anyone can help?

Answer 1

At the end, I just removed the template, and let ES guest the field type. It did work.