Kafka Connect-在写入HDFS接收器之前修改记录

Question

I have installed Kafka connect using confluent-4.0.0 Using hdfs connector I am able to save Avro records received from Kafka topic to hive. I would like to know if there is any way to modify the records before writing into hdfs sink. My requirement is to do small modifications to values of the record. For Example, performing arithmetic operations on integers or manipulation of strings etc. Please suggest if there any way to achieve this

Answer 1

You have several options.

1. Single Message Transforms , which you can see in action here . Great for light-weight changes as messages pass through Connect. Configuration-file based, and extensible using the provided API if there's not an existing transform that does what you want.

   See the discussion here on when SMT are suitable for a given requirement.

2. KSQL is a streaming SQL engine for Kafka. You can use it to modify your streams of data before sending them to HDFS. See this example here .

3. KSQL is built on the Kafka Stream's API , which is a Java library and gives you the power to transform your data as much as you'd like. Here's an example .

Answer 2

Take a look at Kafka connect transformers [1] & [2]. You can build a custom transformer library and use it in connector.

[1] http://kafka.apache.org/documentation.html#connect_transforms [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect