简体   繁体   English

ParDo function 不等待 Apache BEAM 中的 Window

[英]ParDo function not waiting for Window in Apache BEAM

I've created a Window in Apache BEAM to wait either a window to expire or the window is too full (messages are unbounded and coming from a pub/sub queue): I've created a Window in Apache BEAM to wait either a window to expire or the window is too full (messages are unbounded and coming from a pub/sub queue):

   .apply("Window", Window.<TraceUpdateMessage>into(new GlobalWindows())
                        .withTimestampCombiner(TimestampCombiner.EARLIEST)
                        .triggering(Repeatedly.forever(
                                AfterFirst.of(
                                        AfterProcessingTime
                                                .pastFirstElementInPane()
                                                .plusDelayOf(Duration.standardMinutes(options.getFixedWindowDuration())),
                                        AfterPane
                                                .elementCountAtLeast(options.getFixedWindowElementCountTrigger())
                                )
                        ))
                        .withAllowedLateness(Duration.ZERO)
                        .discardingFiredPanes()
                )

I've got a ParDo right after the window:在 window 之后,我有一个 ParDo:

 .apply("Aggregate", ParDo.of(
                        new CustomAggregationFunction()
)

the problem is that the CustomAggregationFunction is getting called right away, and it doesn't look like the window is doing anything.问题是CustomAggregationFunction被立即调用,它看起来不像 window 正在做任何事情。 The goal is go have the CustomAggregationFunction get called only when a Window is full and process all elements inside that window.目标是 go 只有在CustomAggregationFunction已满并处理 window 中的所有元素时才调用 CustomAggregationFunction。

Windowing doesn't take effect until any grouping transform.在任何分组转换之前,窗口化不会生效。 Inserting grouping transform such as group by key or reshuffle between Window and ParDo should work.WindowParDo之间插入分组转换(例如按键分组或重新洗牌)应该可以工作。

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 如何在Java中使用Apache Beam ParDo函数读取JSON文件 - How to read a JSON file using Apache beam parDo function in Java 如何在处理PCollection中的元素时将元素发布到kafka主题 <KV<String,String> &gt;在apache梁中的ParDo功能? - How to publish elements to a kafka topic while processing the elements in the PCollection<KV<String,String>> in ParDo function in apache beam? Apache Beam:如何在使用重复数据删除时解决“ParDo 需要确定性密钥编码器才能使用 state 和计时器”function - Apache Beam: How to solve "ParDo requires a deterministic key coder in order to use state and timers" while using Deduplication function Apache Beam DirectRunner 启用不同 ParDo/DoFn 的多线程处理 - Apache Beam DirectRunner enable multi threaded processing of different ParDo/DoFn Apache 梁的 ID Window Session - ID of Apache Beam Window Session Apache 光束中每小时(顺时针)的窗口数据 - window Data hourly(clockwise) basis in Apache beam 在 Apache Beam 中使用用户会话窗口进行状态处理 - Stateful processing with User Session Window in Apache Beam 如何删除滑动窗口中的重复项 - Apache Beam - How to remove duplicates in sliding window - Apache Beam Apache Beam - 重复数据删除功能的限制是什么 - Apache Beam - what are the limits of Deduplication function 有没有办法在 Beam 的 ParDo 转换中创建 SpecificRecord 列表以编写 Parquet 文件? - Is there a way to create a list of SpecificRecord in a ParDo transformation in Beam for writing Parquet files?
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM