Flink KeyBy 性能

Question

I'm performance benchmarking my Flink application that reads data from Kafka, transforms it and dumps it into another Kafka topic. I need to keep the context so messages with same order-id are not treated as brand new orders. I'm extending RichFlatMapFunction class with ValueState to achieve that. As I understand, I'll need to use KeyStream before I can call flatMap:

env.addSource(source()).keyBy(Order::getId).flatMap(new OrderMapper()).addSink(sink());

The problem is keyBy is taking very long time from my prespective (80 to 200 ms). I say keyBy is taking because if I remove keyBy and replace flatMap with a map function, 90th percentile of latency is about 1ms. Is there a way to use state/context without using keyBy or maybe make keyBy fast somehow?

Answer 1

The keyBy is expensive because it requires a.network shuffle -- every record is serialized, sent to the downstream instance responsible for that key, and then deserialized.

For the pipeline you've described, this is unavoidable. But your choice of serializer can make a big difference.

For more ideas about how to reduce latency, see Flink optimal configuration for minimum Latency .

As for the choice of key, if you need to deduplicate by orderId, then you'll have to key by the orderId.