[英]How to copy and transform all messages from one kafka topic (in avro format) to another topic (in json format)
My team is using Kafka Confluent (enterprise version) and we are very new to kafka.我的团队正在使用 Kafka Confluent(企业版),我们对 kafka 非常陌生。
We have 2 topic A and B.我们有 2 个主题 A 和 B。
Topic A will receive all json messages and formatted by Avro Schema (using URL to schema registry).主题 A 将接收所有 json 消息并由 Avro Schema 格式化(使用 URL 到模式注册表)。
By some reasons, The development tool we are using does not support receiving messages from topic A in avro format.由于某些原因,我们使用的开发工具不支持从主题A接收avro格式的消息。 We create Topic B and want to use KsqlDB to copy all messages from topic A to topic B and also transform all messages from avro format to normal json format so that we can develop a component that can pick up json messages from topic B.我们创建主题 B 并希望使用 KsqlDB 将所有消息从主题 A 复制到主题 B,并将所有消息从 avro 格式转换为正常的 json 格式,以便我们可以开发一个可以从主题 B 中提取 json 消息的组件。
Could you please show me code to create ksql stream to do that.你能否告诉我创建ksql stream 的代码来做到这一点。
Register the inbound Avro data stream注册入站 Avro 数据 stream
CREATE STREAM MY_AVRO_SOURCE
WITH (KAFKA_TOPIC='my_source_topic', FORMAT='AVRO');
Tell ksqlDB to read all messages from the beginning of the topic告诉 ksqlDB 从主题开始读取所有消息
SET 'auto.offset.reset' = 'earliest';
Write all the messages to a new stream in JSON将所有消息写入 JSON 中的新 stream
CREATE STREAM MY_JSON_TARGET
WITH (FORMAT='JSON')
AS SELECT * FROM MY_AVRO_SOURCE;
By default this will populate a target topic called MY_JSON_TARGET
;默认情况下,这将填充一个名为MY_JSON_TARGET
的目标主题; you can specify KAFKA_TOPIC
in the WITH
clause if you want to use a different target topic name.如果要使用不同的目标主题名称,可以在WITH
子句中指定KAFKA_TOPIC
。
If only the value (not key) in the source is Avro then use VALUE_FORMAT
instead of FORMAT
in the WHERE
clause.如果只有源中的值(不是键)是 Avro,则在WHERE
子句中使用VALUE_FORMAT
而不是FORMAT
。
Ref: https://docs.ksqldb.io/en/latest/reference/serialization/参考: https://docs.ksqldb.io/en/latest/reference/serialization/
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.