Tagging the Logs by Logstash - Grok - ElasticSearch

Question

Summary:

I am using Logstash - Grok and elastic search and my main aim is to First accept the logs by logstash, parse them by grok and associate tags with the messages depending on the type of the log, and then finally feed it to the Elastic server to query with Kibana.

I have already written this code but am not able to get the tags in Elastic Search. This is my logstash confif file.

input {
  stdin {
    type => "stdin-type"
  }
}
filter {
  grok {
    tags    => "mytags"
    pattern => "I am a %{USERNAME}"
    add_tag => "mytag"
    named_captures_only => true
  }
}
output {
  stdout { debug => true debug_format => "json"}
  elasticsearch {}
}

Where am I going wrong?

Answer 1

1) I would first start with editing your values to match the data type they represent. For example

     add_tag => "mytag"

actually should have an array as it's value, not a simple string. Change that to

     add_tag => ["mytag"]

as a good start. Double check all your values and verify they are of the correct type for logstash.

2) You are limiting your grok filters to messages that are already tagged with "mytags" based on the config line

    tags => "mytags"

I don't see anywhere where you have added that tag ahead of time. Therefore, none of your messages will even go through your grok filter.

3) Please read the logstash docs carefully. I am rather new to the Logstash/Grok/ES/Kibana etc. world as well, but I have had very similar problems to what you have had, and all of them were solved by paying attention to what the documentation says.

Answer 2

You can run LogStash by hand (You may already be doing this) with /opt/logstash/bin/logstash -f $CONFIG_FILE and can check that your config file is valid with /opt/logstash/bin/logstash -f $CONFIG_FILE --configtest I bet you're already doing that though.

You may need to put your add_tag stanza into an array

grok {
    ...
    add_tag => [ "mytag" ]
}

It could also be that what you're piping into STDIN isn't being matched in the grok pattern. If grok doesn't match is should result in _grokparsefailure being added to your tags. If you see those, it means your grok pattern isn't firing.

A better way to do this may be...

input {
  stdin {
    type => 'stdin'
  }
}
filter {
  if [type] = 'stdin' {
    mutate {
     add_tag => [ "mytag" ]
    }
  }
}
output {
  stdout {
    codec => 'rubydebug'
  }
}

This will add a "mytag" tag to all things coming from standard in, wether they're groked or not.