kafka_2.11-0.8.2.1 Consumer Rebalancing

Question

I have a topic with 50 partitions. I have a single consumer project which has 50 threads processing 50 message streams. As of now , the project is running in only one node. I am using high level consumer. Can the same project be deployed in more than 1 node to achieve re balancing? Are there any issues that I need to be wary of?

Answer 1

Can the same project be deployed in more than 1 node to achieve re balancing?

Yes, You can deploy consumer to more than one node to read from the topic.

Are there any issues that I need to be wary of? Even before going to multiple node you can create multiple consumer on same node to test the re balancing to see what type of issues comes up in configuration. Other than this, you should think about the re balance if one or more node done dies and then other nodes should take care of that.