Elasticsearch is not creating an index received from Logstash Output file

Question

I'm have an Ubuntu 20.04 VM with Elasticsearch, Logstash and Kibana (all rel.7.7.0) What I'm trying to do is (among other things) to have Logstash to receive Syslog and Netflow traps from Cisco devices, transfer them to Elasticsearch and from there to Kibana for visualization.

I created a Logstash config file (cisco.conf) where input and output sections look like this:

input {

 udp {
   port => 5003
   type => "syslog"
     }
 udp {
   port => 2055
   codec => netflow {
   include_flowset_id => true
   enable_metric => true
   versions => [5, 9]
                    }
      }
       }


 output {

stdout { codec => rubydebug }

if [type] == "syslog" {
   elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "ciscosyslog-%{+YYYY.MM.dd}"
   }
 }

 if [type] == "netflow" {
   elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "cisconetflow-%{+YYYY.MM.dd}"
   }
 }

}

The problem is: the index ciscosyslog is created in Elasticsearch with no problem:

$ curl 'localhost:9200/_cat/indices?v'
health status index                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size

yellow open   ciscosyslog-2020.05.21   BRshOOnoQ5CsdVn3l0Z3kw   1   1       1438            0    338.4kb        338.4kb
green  open   .async-search            dpd-HWYJSyW653u7BAhQVg   1   0          2            0     34.1kb         34.1kb
green  open   .kibana_1                xA5PIwKsTHCeOFyj9_NIQA   1   0        111            8    231.9kb        231.9kb
yellow open   ciscosyslog-2020.05.22   kB4vJAooT3-fbIg0dKKt8w   1   1        566            0    159.2kb        159.2kb

However the index cisconetflow is not created as seen in the above table.

I made a debug on Logstash and I can see netflow messages arriving from Cisco devices:

[WARN ] 2020-05-22 17:57:04.999 [[main]>worker1] Dissector - Dissector mapping, field not found in event {"field"=>"message", "event"=>{"host"=>"10.200.8.57", "@timestamp"=>2020-05-22T21:57:04.000Z, "@version"=>"1", "netflow"=>{"l4_src_port"=>443, "version"=>9, "l4_dst_port"=>41252, "src_tos"=>0, "dst_as"=>0, "protocol"=>6, "in_bytes"=>98, "flowset_id"=>256, "src_as"=>0, "ipv4_dst_addr"=>"10.200.8.57", "input_snmp"=>1, "output_snmp"=>4, "ipv4_src_addr"=>"104.244.42.133", "in_pkts"=>1, "flow_seq_num"=>17176}}}
[WARN ] 2020-05-22 17:57:04.999 [[main]>worker1] Dissector - Dissector mapping, field not found in event {"field"=>"message", "event"=>{"host"=>"10.200.8.57", "@timestamp"=>2020-05-22T21:57:04.000Z, "@version"=>"1", "netflow"=>{"l4_src_port"=>443, "version"=>9, "l4_dst_port"=>39536, "src_tos"=>0, "dst_as"=>0, "protocol"=>6, "in_bytes"=>79, "flowset_id"=>256, "src_as"=>0, "ipv4_dst_addr"=>"10.200.8.57", "input_snmp"=>1, "output_snmp"=>4, "ipv4_src_addr"=>"104.18.252.222", "in_pkts"=>1, "flow_seq_num"=>17176}}}
{
          "host" => "10.200.8.57",
    "@timestamp" => 2020-05-22T21:57:04.000Z,
      "@version" => "1",
       "netflow" => {
          "l4_src_port" => 57654,
              "version" => 9,
          "l4_dst_port" => 443,
              "src_tos" => 0,
               "dst_as" => 0,
             "protocol" => 6,
             "in_bytes" => 7150,
           "flowset_id" => 256,
               "src_as" => 0,
        "ipv4_dst_addr" => "104.244.39.20",
           "input_snmp" => 4,
          "output_snmp" => 1,
        "ipv4_src_addr" => "172.16.1.21",
              "in_pkts" => 24,
         "flow_seq_num" => 17176
    }

But at this point I can't tell if Logstash is not delivering the information to ES or if ES is failing to create the index, Current facts are:

a) Netflow traffic is present at Logstash input b) ES is creating only one of the two indexes received from Logstash.

Thanks.

Answer 1

You have conditionals in your output, using the type field, your first input is adding this field with its correct value, but your second input does not have the field, so it will never match your conditional.

Add the line type => "netflow" in your second input as you did with your first one.