kafka-connect-elasticsearch: storing messages as format of predefined index

Question

Example:

{"id":"1","firstName":"abc","lastName":"xyz","dob":"12/09/1995","age":"23"}

This message structure is in kafka topic, but i want to index this in elasticsearch as below

{"id":"1","name"{"firstName":"abc","lastName":"xyz"},"dob":"12/09/1995","age":"23"}

how I can achieve this?

Answer 1

Two options:

1. Stream processing against the data in the Kafka topic. Using Kafka Streams you could wrangle the data model as required. KSQL would work for this in inverse but doesn't support creating STRUCTs yet. Other stream processing options would be Flink, Spark Streaming, etc

2. Modify the data as it passes through Kafka Connect, using Single Message Transform. There's no pre-built transform that does this but you could write one using the API .

Disclaimer: I work for Confluent, the company behind the open-source KSQL project, constributes to Kafka Streams, Kafka Connect, etc.