简体繁体 English

kafka-connect-elasticsearch：如何将 elasticsearch 与消费者组同步？

[英]kafka-connect-elasticsearch: How to sync elasticsearch with consumer group?

原文 2021-05-08 11:41:16 2 1 elasticsearch/ apache-kafka/ apache-kafka-connect/ confluent-platform

I want to query messages in a Kafka topic but not all messages, not from the beginning.我想查询 Kafka 主题中的消息，但不是所有消息，而不是从头开始。 I just need to see which messages are not yet committed based on a consumer group.我只需要根据消费者组查看哪些消息尚未提交。 So, basically what I want to have is to delete the documents whose offset is lower than a consumer group offset.所以，基本上我想要的是删除偏移量低于消费者组偏移量的文档。

At this point, if I use elastic-connector, is there any way or a workaround to delete documents from the elastic index after a message is consumed and committed?此时，如果我使用弹性连接器，是否有任何方法或解决方法可以在消息被消费和提交后从弹性索引中删除文档？

Or, should I use Kafka Streams and how?或者，我应该使用 Kafka Streams 以及如何使用？

1 个解决方案

The sink connector only deletes documents when that property is explicitly enabled and there is a null valued record for a document ID in the topic you're reading.接收器连接器仅在显式启用该属性并且您正在阅读的主题中的文档 ID 存在 null 值记录时删除文档。 This means you need to actually consume this null record and have it be processed by the connector这意味着您需要实际使用此 null 记录并由连接器处理

see which messages are not yet committed查看哪些消息尚未提交

This would imply messages that have not been processed by the connector, making them not searchable in Elasticsearch这意味着连接器尚未处理的消息，使它们无法在 Elasticsearch 中搜索

delete the documents whose offset is lower than a consumer group offset删除偏移量小于消费者组偏移量的文档

If you created a fresh index in Elasticsearch that's only used by the connector, you could pause the connector, then truncate the index , then resume the connector如果您在 Elasticsearch 中创建了仅由连接器使用的新索引，则可以暂停连接器，然后截断索引，然后恢复连接器

is there any way or a workaround to delete documents from the elastic index after a message is consumed and committed在使用和提交消息后，是否有任何方法或解决方法可以从弹性索引中删除文档

Directly use the DELETE API直接使用DELETE API