Can I use Kafka between Logstash and Elasticsearch? ( Using two Kafka)

Question

I'm trying to integrate Apache Kafka with Elastic Stack (Beats, Logstash, Elasticsearch, and Kibana)

From the diagram, Kafka is located between Beats and Logstash. I was wondering if I can put another Kafka between Logstash and Elasticsearch. (Where I drew with a red pen.)

Two Kafka sound okay?

Any ideas or thoughts to share?

Answer 1

Yes.

1. Logstash can write to Kafka as an output.

2. You can use Kafka Connect Elasticsearch for streaming from Kafka into Elasticsearch.

If you want to buffer/scale the output from Logstash by using Kafka here, it is possible and would make sense.

But bear in mind that you could also:

(a) write from Beats to Kafka and do any processing with KSQL/Kafka Streams etc to write back to Kafka and then Kafka Connect to Elasticsearch or (b) just write from Logstash to Elasticsearch