在kafka中为同一主题创建多个使用者

Question

I am newbie and can see one example with one consumer in below github repository, but any ideas how to create multiple consumers for same topic in go lang?

https://github.com/confluentinc/confluent-kafka-go/tree/master/examples

Any consumer factory (to generate N consumers) available in confluent-kafka to read same topic (with partitions)?

Answer 1

There is an example in the Confluent github repo :

https://github.com/confluentinc/confluent-kafka-go/blob/master/examples/consumer_example/consumer_example.go

If you want to create multiple consumers for the same topic, there are two scenarios :

1.Create each consumer with different group id.

c1, err := kafka.NewConsumer(&kafka.ConfigMap{
        "bootstrap.servers":    broker,
        "group.id":             group1,
        "session.timeout.ms":   6000,
        "default.topic.config": kafka.ConfigMap{"auto.offset.reset": "earliest"}})

c2, err := kafka.NewConsumer(&kafka.ConfigMap{
        "bootstrap.servers":    broker,
        "group.id":             group2,
        "session.timeout.ms":   6000,
        "default.topic.config": kafka.ConfigMap{"auto.offset.reset": "earliest"}})

2. If you want multiple consumers for the same topic but with the same group id, it will start consuming from single partition. ie A topic contains 3 partitions and you create 3 consumers with same group id, each consumer will consume from one partition