How to add dynamic hosts in Elasticsearch and logstash

Question

I have prototype working for me with Devices sending logs and then logstash parsing it and putting into elasticsearch.

Logstash output code :-

    output{
    if [type] == "json" {
    elasticsearch {
     hosts => ["host1:9200","host2:9200","host3:9200"]
     index => "index-metrics-%{+xxxx.ww}"
    }
  }

} 

Now My Question is :

I will be producing this solution. For simplicity assume that I have one Cluster and I have right now 5 nodes inside that cluster.

So I know I can give array of 5 nodes IP / Hostname in elasticsearch output plugin and then it will round robin to distribute data.

How can I avoid putting all my node IP / hostnames into logstash config file.

As system goes into production I don't want to manually go into each logstash instance and update these hosts.

What are the best practices one should follow in this case ?

My requirement is :

I want to run my ES cluster and I want to add / remove / update any number of node at any time. I need all of my logstash instances send data irrespective of changes at ES side.

Thanks.

Answer 1

If you want to add/remove/update you will need to run sed or some kind of string replacement before the service startup. Logstash configs are "compiled" and cannot be changed that way.

hosts => [$HOSTS] ... $ HOSTS="\\"host1:9200\\",\\"host2:9200\\"" $ sed "s/\\$HOSTS/$HOSTS/g" $config

Your other option is to use environment variables for the dynamic portion, but that won't allow you to use a dynamic amount of hosts.