如何使用 Logstash 和 grok 插件创建字段

Question

我有以下格式的 tomcat 日志

10.0.6.35 - - [21/Oct/2019:00:00:04 +0000] "GET /rest/V1/productlist/category/4259/ar/final_price/asc/4/20 HTTP/1.1" 200 14970 12

我想创建最后两列的字段，即字节和持续时间，并想使用 Kibana 对其进行分析。 我曾使用 Filebeat 和 Logstash 将数据传输到 Elasticsearch。

我的 Logstash 配置文件如下：

我曾尝试使用以下配置，但无法在 kibana 上看到该字段。

input {
     beats {
     port => 5044
  }
 }

filter {
  grok {
  match => ["message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes}(?m) %{NUMBER:duration}" ]
#match=>{"duration"=> "%{NUMBER:duration}"}
# match => { "message" => "%{COMBINEDAPACHELOG}" }

  }
#  mutate {
#    remove_field => ["@version", "@timestamp"]
#  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
if [fields][log_type] == "access-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
if [fields][log_type] == "application-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
else
{
  elasticsearch {
    hosts => ["172.31.30.73:9200"]
    index => "logstashhh-%{+YYYY.MM.dd}"
}

我希望持续时间和字节成为我在 Kibana 上的可视化字段。

Answer 1

试试这个作为你的logstash配置：

input {
     beats {
     port => 5044
  }
 }

filter {
  grok {
  match => ["message" => "%{NUMBER:bytes}(?m) %{NUMBER:duration}$" ]
#match=>{"duration"=> "%{NUMBER:duration}"}
# match => { "message" => "%{COMBINEDAPACHELOG}" }

  }
#  mutate {
#    remove_field => ["@version", "@timestamp"]
#  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
if [fields][log_type] == "access-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
if [fields][log_type] == "application-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
else
{
  elasticsearch {
    hosts => ["172.31.30.73:9200"]
    index => "logstashhh-%{+YYYY.MM.dd}"
}