Kafka Topic and Partition allocation for consumer

Question

I just started working on kafka, I need to develop a consumer client using sarama go pkg, the client is supposed to be a part of consumer group, and is needed to read from two topic A and B, client needs to read from some partitions of topic A allocated to it by any balance strategy and for B it needs to read from all partition (B is kinda like brodcast topic).

Workflow:

• consumer group xx.
• I have two topic A and B with 6 partition [0,1,2...5] each.
• I have two consumer C1 and C2 in xx, data should be read in such a way:
  C1 reads from A:[0,1,2] and from B:[0,1,2,3,4,5,6]
  C2 reads from A:[3,4,5] and from B:[0,1,2,3,4,5,6]

note: in case an new client is added the partition in A should be rebalance and all partition in B should be read.

I tried implementing my custom balance strategy but failed, Please let me know if this can be done and how to do it.

Answer 1

For any consumer in the same consumer group, it is not possible for multiple to be listening to any overlapping partitions. In other words, listening to all partitions of topic B cannot be done unless you move C2 consumer to its own unique group, regardless of the rebalancing strategy for consumer groups for topic A.

Answer 2

You need to implement both Partition Consumer and Group Consumer in your service.

1. Group Consumer

Use Group Consumer to consume messages from topic "A". You can implement ConsumerGroup interface in Sarama library. Both your consumers "C1" and "C2" need to subscribe to topic "A" as a group (using the same client ID).

2. Partition Consumer

Use Partition consumers to consume messages in topic "B". Use Consumer interface in Sarama library for this. "C1" and "C2" need to subscribe to all partitions in starting up and also after a rebalance.