kafka-streams：在将消息结构发送到kafka主题之前修改消息结构

Question

I have a scenario where I need to change the message structure which I am getting from JDBC source connector before it hits to kafka . I am a beginner to kafka so how kafka streams can help me in achieving this .

Input -

{"id":"123","firstName":"pqr","lastName":"xyz","age":"23","dob":"20-09-1995"} 

I want to store this in elasticseacrh index as

{"id":"123","name":{"firstName":"pqr","lastName":"xyz"},"age":"23","dob":"20-09-1995"} 

I am not set on kafka streams, I thought to use it, if using KSQL is possible i want to know that.

Answer 1

It seems you are using Kafka Connect. For this case, "Single Message Transform" (SMT) are your friend. Those allows you to transform each message one-by-one before the connector writes the data into the topic.

Check out this blog post for more details: https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/