[英]How to send the windowed aggregations result to the output topic only when window is finished?
I am facing an issue while doing the window aggregations.我在进行窗口聚合时遇到了一个问题。 I want to perform sum of the values per key and the result is sent to the output topic only when window is finished.我想对每个键的值进行求和,并且仅在窗口完成时将结果发送到输出主题。 The problem is that every event in "input" topic will produce an event to "output" topic.问题是“输入”主题中的每个事件都会产生一个“输出”主题的事件。 I would like to publish an event to the output topic only when a window is finished.我想仅在窗口完成时将事件发布到输出主题。 For example if the window is of one minute, send a single event per key per minute.例如,如果窗口为一分钟,则每分钟每个键发送一个事件。 The sample code is as follows:示例代码如下:
.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(2))
.reduce((v1, v2) -> String.valueOf(Integer.parseInt(v1) + Integer.parseInt(v2)))
.suppress(Suppressed.untilWindowCloses(BufferConfig.unbounded()))
.toStream((k,v) -> k.key())
.to("output_topic");
But I am getting the following exception:但我收到以下异常:
Exception in thread "learningtime_application-665cd31a-1957-448b-8cf7-779ab359cfd2-StreamThread-1" org.apache.kafka.streams.errors.ProcessorStateException: task [1_0] Failed to flush state store KSTREAM-REDUCE-STATE-STORE-0000000003 Caused by: java.lang.ClassCastException: class org.apache.kafka.streams.kstream.Windowed cannot be cast to class java.lang.String (org.apache.kafka.streams.kstream.Windowed is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')线程“learningtime_application-665cd31a-1957-448b-8cf7-779ab359cfd2-StreamThread-1”中的异常 org.apache.kafka.streams.errors.ProcessorStateException:任务 [1_0] 无法刷新状态存储 KSTREAM-REDUCE-00003000000引起:java.lang.ClassCastException: class org.apache.kafka.streams.kstream.Windowed 不能转换为 class java.lang.String (org.apache.kafka.streams.kstream.Windowed is in unnamed module of loader ' app'; java.lang.String 位于加载器 'bootstrap' 的模块 java.base 中)
You are hitting a known bug: https://issues.apache.org/jira/browse/KAFKA-9259您遇到了一个已知错误: https : //issues.apache.org/jira/browse/KAFKA-9259
The suppress
operator does not correctly pick up the default serdes from the config, ie, it uses the key serde without converting it to windowed-key-serdes. suppress
运算符没有正确地从配置中选择默认的 serdes,即它使用密钥 serde 而不将其转换为 windowed-key-serdes。
As a workaround, you need to specify the serdes explicitly in reduce()
via Materialized.with(...)
.作为一种解决方法,您需要通过Materialized.with(...)
在reduce()
明确指定 serdes。 You pass in plain key and value serdes, and reduce
will convert the key-serde into a windowed-key-serde that will than also be passed into suppress()
.您传入普通的键和值 serdes, reduce
会将键 serde 转换为窗口键 serde,然后也将传递给suppress()
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.