[英]ParDo function not waiting for Window in Apache BEAM
I've created a Window in Apache BEAM to wait either a window to expire or the window is too full (messages are unbounded and coming from a pub/sub queue): I've created a Window in Apache BEAM to wait either a window to expire or the window is too full (messages are unbounded and coming from a pub/sub queue):
.apply("Window", Window.<TraceUpdateMessage>into(new GlobalWindows())
.withTimestampCombiner(TimestampCombiner.EARLIEST)
.triggering(Repeatedly.forever(
AfterFirst.of(
AfterProcessingTime
.pastFirstElementInPane()
.plusDelayOf(Duration.standardMinutes(options.getFixedWindowDuration())),
AfterPane
.elementCountAtLeast(options.getFixedWindowElementCountTrigger())
)
))
.withAllowedLateness(Duration.ZERO)
.discardingFiredPanes()
)
I've got a ParDo right after the window:在 window 之后,我有一个 ParDo:
.apply("Aggregate", ParDo.of(
new CustomAggregationFunction()
)
the problem is that the CustomAggregationFunction
is getting called right away, and it doesn't look like the window is doing anything.问题是CustomAggregationFunction
被立即调用,它看起来不像 window 正在做任何事情。 The goal is go have the CustomAggregationFunction
get called only when a Window is full and process all elements inside that window.目标是 go 只有在CustomAggregationFunction
已满并处理 window 中的所有元素时才调用 CustomAggregationFunction。
Windowing doesn't take effect until any grouping transform.在任何分组转换之前,窗口化不会生效。 Inserting grouping transform such as group by key or reshuffle between Window
and ParDo
should work.在Window
和ParDo
之间插入分组转换(例如按键分组或重新洗牌)应该可以工作。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.