Kafka中每个主题的一个消费者线程针对多个分区是否会导致延迟？

Question

Our kafka setup is as follows:

1. 30 partitions per topic
2. 1 consumer thread

we configured this way to be able to scale-up in the future.

we wanted to minimize the times we re-balance when we need to scale-up by adding partitions because latency is very important to us and during re-balances messages can be stuck till the coordination phase is done

Having 1 consumer thread with many partitions per 1 topic can effect somehow the overall messaging consuming latency?

Answer 1

More partitions in a Kafka cluster leads to higher throughput however, you need to be aware that the number of partitions has an impact on availability and latency as well.

In general more partitions,

• Lead to Higher Throughput
• Require More Open File Handles
• May Increase Unavailability
• May Increase End-to-end Latency
• May Require More Memory In the Client

You need to study the trade-offs and make sure that you've picked the number of partitions that satisfies your requirements regarding throughput, latency and required resources.

For further details refer to this blog post from Confluent.

My opinion: Make some tests and write down your findings. For example, try to run a single consumer over a topic with 5, 10, 15, ... partitions, measure the impact and pick the configuration that meets your requirements. Finally ask yourself if you will ever need x partitions. At the end of the day, if you need more partitions you should not worry about re-balancing etc. Kafka was designed to be scalable .