Flink 连接键控和非键控流

Question

What happens if I connect keyed and non keyed streams in Flink? For example;

val stream1 = source.keyBy(_.customerId)
val stream2 = source2.map(operation()...)

stream1.connect(stream2).process(processData).addSink(kafkaSink)

The situation here is that I'm trying to apply set of rules (non keyed stream, stream2) to another stream (stream1), I'm normally broadcasting rules stream but the size of the broadcast is getting bigger and bigger. So, I'd just like to learn that what happens if I don't broadcast one of the streams and connect them in terms of parallelism, state etc.?

Answer 1

When you connect two streams, they must fall into one of these cases:

1. One of the streams is broadcast
2. Neither stream is keyed
3. Both streams are keyed into the same keyspace

You cannot connect a keyed stream to a non-keyed stream, because the resulting connection won't be key-partitioned.