简体繁体中英

Kafka streams: topic partitions->consumer 1:1 mapping not happening

原文 2018-08-31 02:09:00 8 1 apache-kafka/ apache-kafka-streams

I have written a streams application to talk to topic on cluster of 5 brokers with 10 partitions. I have tried multiple combinations here like 10 application instances (on 10 different machines) with 1 stream thread each, 5 instances with 2 threads each. But for some reason, when I check in kafka manager, the 1:1 mapping between partition and stream thread is not happening. Some of the threads are picking up 2 partitions while some picking up none. Can you please help me with same?? All threads are part of same group and subscribed to only one topic. The kafka streams version we are using is 0.11.0.2 and broker version is 0.10.0.2

Thanks for your help

1 answers

Maybe you are hitting https://issues.apache.org/jira/browse/KAFKA-7144 -- I would recommend to upgrade to the latest versions.

Note: you do not need to upgrade your brokers

Kafka - Topic & Partitions & Consumer

Kafka Streams: Internal topic partitions

Kafka Streams: Reading from topic with multiple partitions

changelog topic in Kafka streams - setting or altering partitions

Kafka Consumer get assigned partitions for a specific topic

Kafka consumer - what's the relation of consumer processes and threads with topic partitions

How to assign all partitions of a kafka topic to a consumer in Python3.6?

Kafka topic is still associating with old consumer group of Kafka Streams application

Kafka Should Number of Consumer Threads equal number of Topic Partitions

consumer.poll not fetching records from Kafka Topic having 3 partitions

暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Kafka - Topic & Partitions & Consumer Kafka Streams: Internal topic partitions Kafka Streams: Reading from topic with multiple partitions changelog topic in Kafka streams - setting or altering partitions Kafka Consumer get assigned partitions for a specific topic Kafka consumer - what's the relation of consumer processes and threads with topic partitions How to assign all partitions of a kafka topic to a consumer in Python3.6? Kafka topic is still associating with old consumer group of Kafka Streams application Kafka Should Number of Consumer Threads equal number of Topic Partitions consumer.poll not fetching records from Kafka Topic having 3 partitions

Related Tags

Kafka streams: topic partitions->consumer 1:1 mapping not happening

Question

1 answers

solution1 0 2018-09-02 13:27:24

solution1
0 2018-09-02 13:27:24