使用 Kafka-Connect 将消息路由到不同的 ElasticSearchIndex

Question

I have a kafka topic on which multiple types of messages (all are JSON) are being published to it with header property as :- eventType = "abc" or eventType = "xyz" or eventType = "def" etc.

My requirement is to send these messages from Kafka topic to different elasticSearch index based on eventType header field being sent in kafka. eg if eventType = "abc", push data to elasticsearch index = "abc123" if eventType = "xyz", push data to elasticsearch index = "xyz123"

I am planning to use kafka connect for this operation. I need help how can i achieve this using kafka-connect ?

I am using kafka 1.1.0 and planning to use confluentinc-kafka-connect-elasticsearch--plugin.

Here is my elastic search sink connect config file :-

name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=test-elasticsearch-sink
topic.index.map=test-1:elastic_purchase_index1
key.ignore=true
connection.url=<url>
type.name=purchasev1
key.ignore=true
schema.ignore=true

Answer 1

based on eventType header field

Use ExtractTopic$Value + RegexRouter to route messages to individual indicies.

Then use topics.regex in the ElasticSearch sink to define patterns for the topics to read from