How to dynamically add more Kafka topics to Logstash configuration

Question

Is there any option how to add to logstash kafka input multiple kafka topics? I am finding dynamic solution cause number of my topics are changing.

my logstash config looks like :

input {
    kafka {
    bootstrap_servers => "localhost:9092"
    topics => ["company0","company1","company2","company3","company4"]
    }
} 
output {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "indextry"
      workers => 1
    }
}

in this solution I am using 5 kafka topics but in another case I want to use 20 for example.

Is there any dynamic solution how to add kafka topics here?

Answer 1

Logstash kafka input supports the use of a topics_pattern configuration.

kafka {
    bootstrap_servers => "localhost:9092"
    topics_pattern => ["company.*"]
}

This config will consume every topic that starts with "company".