Flink State 跨多个Transformer

Question

How can I access a state using the same-id across multiple transformers, for example the following stores an Order object via ValueState in OrderMapper class:

env.addSource(source1()).keyBy(Order::getId).flatMap(new OrderMapper()).addSink(sink1());

Now I would like to access the same Order object via a SubOrderMapper class:

env.addSource(source2()).keyBy(SubOrder::getOrderId).flatMap(new SubOrderMapper()).addSink(sink2());

Edit: Looks like it's not possible to have state maintained across multiple operators, is there a way to have one operator accept multiple inputs, lets say 5 sources?

Answer 1

Take a look at CoProcessFunction

To realize low-level operations on two inputs, applications can use CoProcessFunction or KeyedCoProcessFunction. This function is bound to two different inputs and gets individual calls to processElement1(...) and processElement2(...) for records from the two different inputs.

Also side outputs might be useful for you. side output

Edit: Union operator my be an option. Union

Answer 2

You can create a custom EitherOfFive class that contains one of your five different stream values (I'm assuming they are all different). See Flink's Either class for the one of two case.

Each input stream would use a Map function that converts the input class type to an EitherOfFive type.

There would be a getKey() method that would figure out (based on which of the five values is actually set) what key to return. And then you can have a single KeyedProcessFunction that takes as input this EitherOfFive type.

If the output is always the same, then you're all set. Otherwise you'll want side outputs, one per type, that feed the five different sinks.