ElasticSearch to Kafka event - on each change using Logstash

Question

End goal: To connect Elasticsearch and kafka and recieve ongoing change event in ES indexes to kafka. From kafka, I have listeners to do further processing.

Method: I used Logstash input and output plugins. Here is the configuration code.

input {
        elasticsearch {
                hosts => ["localhost:9200"]
                index => "liferay-20116"
        }
}
output {
      kafka {
        topic_id => "elastic-topic-index"
        codec => json
      }
}

It is working but there is one weird issue.

When I listen to kafka it reads all the document from ES which is around 176 docs right now.

Once it reads, it stops for a while say 2 seconds and then read whole 176 docs again!

I don't know what is the issue, is this due to Logstash behavior or Kafka is acting weird?

Any help would be appreciated!

Answer 1

This is standard behavior of this plugin - it push data matching to given query. If you want to have only changed documents, the only workaround is to build this knowledge of what changed yourself - for example, you need to have timestamp for entries, and then incorporate this knowledge into query that is sent to ES.

Answer 2

I did figure it out without using Logstash.

I am using ElasticSearch plugin which emits events on every change as web socket.

Reference: https://codeforgeek.com/2017/10/elasticsearch-change-feed/