Filebeat is not sending data to Logstash

Question

I'm trying to send alerts from Snort IDS to Elasticsearch, therefore I'm using 3 technologies:

• Elasticsearch- https://pastebin.com/uCNMaZFJ
• Logstash- https://pastebin.com/zgnbbw9K
• Filebeat- https://pastebin.com/45rC3rW5

My filebeat configuration file has this code inside:

input {
beats {
    port => 5044
}

} filter {

if [type] == "snort" {

    # parse the message into individual fields
    grok {
        match => { "message" => "(?<ts>.*\d{2}:\d{2}:\d{2})\s(?<host>.*?)\s.*?\s\[(?<generator_id>.*?)::(?<signature_id>.*?):.*?\]\s(?<signature>.*?)\s\[Classification:\s(?<classification>.*?)\]\s\[Priority:\s(?<priority>.*?)\].*?{(?<protocol>.*?)\}\s(?<source_ip>.*?):(?<source_port>.*?)\s-\>\s(?<destination_ip>.*?):(?<destination_port>.*)" }
    }

    # remove the original message if parsing was successful
    if !("_grokparsefailure" in [tags]) {
        mutate {
            remove_field => [ "message" ]
        }
    }

    # parse the timestamp and save in a new datetime field
    if [ts] {
        date {
            match => [ "ts", "MMM dd HH:mm:ss" ]
            target => "sys_timestamp"
        }

        # remove the original timestamp if date parsing was successful
        if !("_dateparsefailure" in [tags]) {
            mutate {
                remove_field => [ "ts" ]
            }
        }
    }
}

} output {

# save events to Elasticsearch with the uuid as the document id
elasticsearch {
    hosts => ["localhost:9200"]
manage_template => false
    index => "teste-%{+YYYY-MM-dd}"
}

}

I am expecting to see snort's alert logs when I check " http://localhost:9200/ola- */_search?pretty", however the alerts are not retrieved. I'm struggling to fix this problem...I don't have any idea what is the problem.

Thanks in advance!

Answer 1

What is the version of your stack? Your filebeat configuration file has both filebeat.prospectors and filebeat.inputs , since version 6.3 you should use filebeat.inputs instead of filebeat.prospectors .

Also the document_type configuration was removed since version 6.0, your message probably does not have a field called type with a value of snort , which is your main filter in the logstash pipeline. It's better to filter your messages using tags.

Use this in your filebeat.yml instead.

filebeat.inputs:
- type: log
  paths:
    - /var/log/snort/*.log
  tags: ["snort"]

And change your logstash filter, just use if "snort" in [tags] instead of if [type] == "snort"

Your output is sending any message that you receives to an index called teste-%{+YYYY-MM-dd} , why are you running a search against an index called ola-* ? You should run a search against the teste-* index.

I recommend that you run your pipeline with the stdout output to see what is happening.

Just put this on your pipeline to see if you are getting any message and how are those messages.

output {
  stdout { }
}