Is using Kafka as an input source for Flink a performance bottleneck?

Question

Flink允许阅读Kafka主题，这是性能瓶颈使Flink整体变慢吗？

Answer 1

Kafka partitions can scale horizontally to accomodate for higher thoughput.

One Flink consumer thread can only be assigned to one Kafka partition.

So, if you have only 1 Kafka partition, and N+1 Flink executors, then you will have N idle tasks, which could be a bottleneck, sure, but that is a tradeoff of having total-ordering within a Kafka topic, not necessarily a Flink problem.

Otherwise, you would create your Kafka topics with ten to hundreds of partitions, and Flink would be fine to consume it.

Answer 2

It depends. If, for example, you have a slow network connection between your Kafka cluster and your Flink cluster, then that will become a performance bottleneck.