用于Kafka Connect的Elasticsearch连接器-偏移量和时间戳

Question

I am using elasticsearch connector (confluent) for kafka connect. I am getting messages from topic into elasticsearch index. I can see that offset is part of _id if no key is used. I want to browse the messages in elasticseach based on offsets and timestamp.

Is it possible to get the offset and timestamp of message as fields in the index?

Here is the elasticsearch document

{ "_index": "test-elasticsearch-sink", "_type": "kafka-connect", "_id": "test-elasticsearch-sink+0+0", "_score": 1, "_source": { "f1": "value1" } }

Thanks, Rajesh

Answer 1

目前尚不可能，但是我们正在考虑一项功能，该功能将允许提升记录元数据，例如timestamp ， topic ， partition ，对字段的offset 。

Answer 2

Using Single Message Transforms you can indeed add new fields including values such as offset and timestamp. Have a look at org.apache.kafka.connect.transforms.InsertField .