[英]Kafka Streams: How to transform message before publishing to different topics
Messages on Topic A are read using Kafka Streams. 使用Kafka Streams阅读有关主题A的消息。 Based on some validation, the message is published to either Topic B or Topic C. This was easily achieved using Kafka Streams' branch method 根据一些验证,该消息将发布到主题B或主题C。使用Kafka Streams的分支方法很容易实现
KStreamBuilder kStreamBuilder = new KStreamBuilder();
KStream<String, String> kStream = kStreamBuilder.stream(Serdes.String(), Serdes.String(), topicA.split(","));
KStream<String, String>[] splitStreams = kStream.branch(
(key, value) -> process(value), //Process method is expensive
(key, value) -> true
);
splitStreams[0].to(Serdes.String(), Serdes.String(), topicB);
splitStreams[1].to(Serdes.String(), Serdes.String(), topicC);
Problem: Based on whether the message passes or fails validation, it is to be edited/appended before publishing it to either Topic B or Topic C. How do I achieve that? 问题:根据消息是否通过验证,在将消息发布到主题B或主题C之前,必须对其进行编辑/附加。如何实现?
for transforming messages you could use the following KStream
methods: 要转换消息,可以使用以下KStream
方法:
map
(for updating message based by key & value) map
(用于根据键和值更新消息) mapValues
(for updating message based by value) mapValues
(用于根据值更新消息) selectKey
(for updating key based by key & value) selectKey
(用于根据键和值更新键) example: 例:
splitStreams[0].mapValues(value -> transformValueForTopicB(value))
.to(Serdes.String(), Serdes.String(), topicB);
splitStreams[1].map((key, value) -> transformValueForTopicC(key, value))
.to(Serdes.String(), Serdes.String(), topicC);
also as I see from your process
method name inside branch
method, potentially you try to process value, but you only need to provide Predicate
(that returns true
or false
) without any processing. 正如我从branch
方法中的process
方法名称中看到的那样,可能会尝试处理值,但只需要提供Predicate
(返回true
或false
)就无需任何处理。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.