How to configure the Kafka Cluster to work with Elastic Search Cluster?

Question

I have to build a log-cluster and monitoring cluster ( For high-availability ) like this topology. I'm wondering to know how to config those log-shippers clusters. ( I have 2 Topo in the Image)

1. If I use Kafka with FileBeat in Kafka Cluster, Will Elastic Search receive duplication data because Kafka has replicas in data?

2. If I use Logstash (In Elastic Search Cluster) for getting logs from Kafka Cluster, how the config should be because I think that Logstash will not know where to read the log efficiency on Kafka Cluster.

Cluster topology

Thanks for reading. If you have any idea, please discuss with me ^^!

Answer 1

As i see both configurations are compatible with Kafka, you can use filebeat, logstash or mixed them in consumer and producer stages!

IMHO all depends about your needs, ie: sometimes we use some filters to rich the data before ingest to kafka (producer stage), or before index the data to elastic (consumer stage), in this case is better work with logsatsh, because is easier using filters than in filebeat

But if you want to play with raw data, maybe filebeat is betther, because the agent is lighter.

About your questions:

1. Kafka has the data replicted, but for HA propouses, you only read one time the data with the same consumer group
2. For read the log from kafka with logstash, you can use the logstash input plugin for kafka, is easy and works fine!

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html