kafka-connect-elasticsearch: How to sync elasticsearch with consumer group?

Question

I want to query messages in a Kafka topic but not all messages, not from the beginning. I just need to see which messages are not yet committed based on a consumer group. So, basically what I want to have is to delete the documents whose offset is lower than a consumer group offset.

At this point, if I use elastic-connector, is there any way or a workaround to delete documents from the elastic index after a message is consumed and committed?

Or, should I use Kafka Streams and how?

Answer 1

The sink connector only deletes documents when that property is explicitly enabled and there is a null valued record for a document ID in the topic you're reading. This means you need to actually consume this null record and have it be processed by the connector

see which messages are not yet committed

This would imply messages that have not been processed by the connector, making them not searchable in Elasticsearch

delete the documents whose offset is lower than a consumer group offset

If you created a fresh index in Elasticsearch that's only used by the connector, you could pause the connector, then truncate the index , then resume the connector

is there any way or a workaround to delete documents from the elastic index after a message is consumed and committed

Directly use the DELETE API