How to remove 'before' key from payload generated by debezium event for updates in SQL server

Question

For every update in SQL server, debezium generates event payload with 'after' and 'before'. I want to get rid of 'before' without flattening the payload.

Use case :

I am using debezium with Kafka and hudi and have created a data lake, I don't need delete operations so I have skipped delete operations.

So I want to reduce the size of payload to half by eliminating the 'before' part so how we can achieve this?

TIA

Answer 1

The only way to remove after with Debezium connector is to flatten . You can "add back" the event metadata using add.fields config.

Alternative solutions include using Kafka Streams/KSQL/Spark/Flink etc. to select/expand the individual fields you're interested in from the Debezium source topic, write to an intermediate topic with the data you want to sink, then have the Hudi consumer read that topic.

Answer 2

Found a solution, used existing transforms. Following transformation works like a charm

transforms=ReplaceField transforms.ReplaceField.type=org.apache.kafka.connect.transforms.ReplaceField$Value transforms.ReplaceField.exclude=before