Prioritising message in Logstash and Kafka

Question

We will be feeding different logs from different sources to kafka from logstash. There would be some particular set(or some format)messages that consumers of kafka will be more interested than other messages. How do we prioritise the messages?

One thought is to create two topics in kafka - 1) high_priority_topic 2) low_priority_topic and then use filter in logstash to push interested messages to high_priority_topic and rest of the messages to other. Is this possible?

Are there any recommended approach for this?

Answer 1

Use either tags or type in logstash to differentiate between high_priority and low_priority, then have two kafka outputs.

if CONDITIONAL {
  kafka {
    topic => "high_priority_topic"
    ...
  }
}
else {
  kafka {
    topic => "low_priority_topic"
    ...
  }
}

Using type the CONDITIONAL would look like

[type] == "high_priority"

Using tagging it would look like

"high_priority" in [tags]