Apache Beam:: 无法与 session window 一起使用 groupbykey 工作

Question

I've a simple problem. Lets say I'm reading a parquet file which produces an avro GenericRecord object per row, as below.

{"name":"john", "surename":"doe", "age":40, "user_pk":"john:doe:40", "unique_attribute":"j1"}
{"name":"john", "surename":"doe", "age":40, "user_pk":"john:doe:40", "unique_attribute":"j2"}
{"name":"john", "surename":"doe", "age":40, "user_pk":"john:doe:40", "unique_attribute":"j3"}
{"name":"john", "surename":"doe", "age":40, "user_pk":"john:doe:40", "unique_attribute":"j4"}

{"name":"paul", "surename":"carl", "age":28, "user_pk":"paul:carl:28", "unique_attribute":"p1"}
{"name":"paul", "surename":"carl", "age":28, "user_pk":"paul:carl:28", "unique_attribute":"p2"}
{"name":"paul", "surename":"carl", "age":28, "user_pk":"paul:carl:28", "unique_attribute":"p3"}

this file was flatten on purpose and I would like to un-flatten them.

• we know that the input is ordered and I would like to process them until the next session key, and pass to the next apply in the pipeline, to keep the memory requirement minimal, so the intermediary stage should return KV<String, Iterable<GenericRecord>> or even better combined KV<String, GenericRecord> .

<"john:doe:40", {"name":"john", "surename":"doe", "age":40, ["unique_attribute":"j1", ...]}>
<"paul:carl:28", {"name":"paul", "surename":"carl", "age":28, "user_pk":, ["unique_attribute":"p1", ...]}

this is what I've got so far;

        pipeline.apply("FilePattern", FileIO.match().filepattern(PARQUET_FILE_PATTERN))
                .apply("FileReadMatches", FileIO.readMatches())
                .apply("ParquetReadFiles", ParquetIO.readFiles(schema))
                .apply("SetKeyValuePK", WithKeys.of(input -> AvroSupport.of(input).extractString("user_pk").get())).setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(schema)))
                .apply(Window.into(Sessions.withGapDuration(Duration.standardSeconds(5L)))).setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(schema)))
                .apply("SetGroupByPK", GroupByKey.create()).setCoder(KvCoder.of(StringUtf8Coder.of(), IterableCoder.of(AvroCoder.of(schema))))
...
...

I don't know if there is a better way of doing it but for now I've used Sessions.withGapDuration windowing strategy. I expected I would get a grouped element KV<String, Iterable<GenericRecord>> element in every ~5seconds, but I'm not getting anything after GroupByKey , I'm not even sure if GroupByKey is actually doing anything, but I know that memory is increasing rapidly so it must be waiting for all the items.

So the question is, how would you setup a windowing function that will allow me to groupbykey. I've also tried Combine.byKey , as it is suppose to be GroupByKey + Windowing Function but couldn't implemented?

Answer 1

I've managed to get the groupby working, but not sure if I understand fully. I had to add two thinks. First one (any?) IO operations in Beam doesn't add timestamp.

.apply("WithTimestamp", WithTimestamps.of(input -> Instant.now()))

second I've added a Triger so the GroupByKey would actually get triggered. No idea why it wasn't triggering in the first place. I'm sure someone has an explanation for this.

.apply("SessionWindow", Window.<KV<String, GenericRecord>>into(Sessions.withGapDuration(Duration.standardSeconds(5L))).triggering(
                        AfterWatermark.pastEndOfWindow()
                                .withLateFirings(AfterProcessingTime
                                        .pastFirstElementInPane().plusDelayOf(Duration.ZERO)))
                        .withAllowedLateness(Duration.ZERO)
                        .discardingFiredPanes())

It is not perfect, still had to wait couple of minutes before I see that the GroupByKey gets triggered even though the window is only 5s , but it gets triggered in the end, which is progress.

EDIT: ok it looks like timestamp wasn't needed, I'm assuming because the window is session based and not time based. I've also change the setting to streaming

        options.as(StreamingOptions.class)
                .setStreaming(true);

I hope this helps to someone who is having a similar issues.