Elasticsearch 未创建从 Logstash Output 文件接收的索引

Question

我有一个 Ubuntu 20.04 VM 与 Elasticsearch、Logstash 和 Kibana（所有 rel.7.7.0）我想要做的是（除其他外）让 Logstash 从 Cisco 设备接收 Syslog 和 Netflow 陷阱到 Elasticsearch 并从那里到 Kibana 进行可视化。

我创建了一个 Logstash 配置文件 (cisco.conf)，其中输入和 output 部分如下所示：

input {

 udp {
   port => 5003
   type => "syslog"
     }
 udp {
   port => 2055
   codec => netflow {
   include_flowset_id => true
   enable_metric => true
   versions => [5, 9]
                    }
      }
       }


 output {

stdout { codec => rubydebug }

if [type] == "syslog" {
   elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "ciscosyslog-%{+YYYY.MM.dd}"
   }
 }

 if [type] == "netflow" {
   elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "cisconetflow-%{+YYYY.MM.dd}"
   }
 }

}

问题是：索引ciscosyslog在Elasticsearch中创建没有问题：

$ curl 'localhost:9200/_cat/indices?v'
health status index                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size

yellow open   ciscosyslog-2020.05.21   BRshOOnoQ5CsdVn3l0Z3kw   1   1       1438            0    338.4kb        338.4kb
green  open   .async-search            dpd-HWYJSyW653u7BAhQVg   1   0          2            0     34.1kb         34.1kb
green  open   .kibana_1                xA5PIwKsTHCeOFyj9_NIQA   1   0        111            8    231.9kb        231.9kb
yellow open   ciscosyslog-2020.05.22   kB4vJAooT3-fbIg0dKKt8w   1   1        566            0    159.2kb        159.2kb

但是，如上表所示，未创建索引 cisconetflow。

我在 Logstash 上进行了调试，我可以看到来自 Cisco 设备的 netflow 消息：

[WARN ] 2020-05-22 17:57:04.999 [[main]>worker1] Dissector - Dissector mapping, field not found in event {"field"=>"message", "event"=>{"host"=>"10.200.8.57", "@timestamp"=>2020-05-22T21:57:04.000Z, "@version"=>"1", "netflow"=>{"l4_src_port"=>443, "version"=>9, "l4_dst_port"=>41252, "src_tos"=>0, "dst_as"=>0, "protocol"=>6, "in_bytes"=>98, "flowset_id"=>256, "src_as"=>0, "ipv4_dst_addr"=>"10.200.8.57", "input_snmp"=>1, "output_snmp"=>4, "ipv4_src_addr"=>"104.244.42.133", "in_pkts"=>1, "flow_seq_num"=>17176}}}
[WARN ] 2020-05-22 17:57:04.999 [[main]>worker1] Dissector - Dissector mapping, field not found in event {"field"=>"message", "event"=>{"host"=>"10.200.8.57", "@timestamp"=>2020-05-22T21:57:04.000Z, "@version"=>"1", "netflow"=>{"l4_src_port"=>443, "version"=>9, "l4_dst_port"=>39536, "src_tos"=>0, "dst_as"=>0, "protocol"=>6, "in_bytes"=>79, "flowset_id"=>256, "src_as"=>0, "ipv4_dst_addr"=>"10.200.8.57", "input_snmp"=>1, "output_snmp"=>4, "ipv4_src_addr"=>"104.18.252.222", "in_pkts"=>1, "flow_seq_num"=>17176}}}
{
          "host" => "10.200.8.57",
    "@timestamp" => 2020-05-22T21:57:04.000Z,
      "@version" => "1",
       "netflow" => {
          "l4_src_port" => 57654,
              "version" => 9,
          "l4_dst_port" => 443,
              "src_tos" => 0,
               "dst_as" => 0,
             "protocol" => 6,
             "in_bytes" => 7150,
           "flowset_id" => 256,
               "src_as" => 0,
        "ipv4_dst_addr" => "104.244.39.20",
           "input_snmp" => 4,
          "output_snmp" => 1,
        "ipv4_src_addr" => "172.16.1.21",
              "in_pkts" => 24,
         "flow_seq_num" => 17176
    }

但此时我无法判断 Logstash 是否没有将信息传递给 ES，或者 ES 是否无法创建索引，目前的事实是：

a) Netflow 流量存在于 Logstash 输入 b) ES 仅创建从 Logstash 接收的两个索引之一。

谢谢。

Answer 1

您在 output 中有条件，使用type字段，您的第一个输入是使用正确值添加此字段，但您的第二个输入没有该字段，因此它永远不会匹配您的条件。

与第一个输入一样，在第二个输入中添加 line type => "netflow" 。