Segregating and pushing Logs into different indices in elasticsearch using logstash based on loglevels

Question

I am new to ELK stack and currently i want to push logs in different elasticsearch indices based on their loglevels. For example lets say logstash-error_logs index will contain only error logs (logs with loglevel as ERROR), logstash-ok_logs index will contain all INFO logs ()logs with loglevel as INFO ) and so on. Given is the format of my logfile:

[2016-01-06 13:29:49] staging.INFO: Callback sent

Hence the grok filter i am using is as given below:

grok {
            match => { "message" => "%{SYSLOG5424SD} %{JAVACLASS:LOGLEVEL}" }
    }

This provides me an additional field of "LOGLEVEL" in the JSON of the logs . Using this field I can use conditions to route the logs to different indices depending upon the loglevel. Hence I am using the following code in the output filter to route the logs accordingly:

output {
    if [LOGLEVEL] in  ["staging.ERROR"] {
            elasticsearch { hosts => ["localhost:9200"]
                            index => "logstash-error_logs"
            }
    }
    else if [LOGLEVEL] in  ["staging.INFO"] {
            elasticsearch { hosts => ["localhost:9200"]
                            index => "logstash-ok_logs"
            }
    }
    else {
            elasticsearch { hosts => ["localhost:9200"]
                            index => "tech"
            }
    }
    stdout {    codec =>     rubydebug }

}

But unfortunately staging. INFO logs are not getting routed to required index.

Answer 1

Two thoughts:

1. "in" is more for tags. Try this:

   if [LOGLEVEL] == "staging.ERROR" { ... }

2. rather than repeating your output stanzas with a small difference in between them, set and use a variable (Note: pseudo-code; I always get the => syntax wrong):

   if [LOGLEVEL] == "staging.ERROR" { mutate { add_field => { "index_name" => "error" } } }

and then in your output:

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-%{index_name}_logs"
    }
}

If you don't want the "index_name" field to be sent to elasticsearch, you can put it in @metadata instead.

Hope that helps.