如何为具有多个分区的 Kafka 主题启动多个使用者？

Question

I have recently started using spark and have to deal with a case where I need to consume multiple partitions of a Kafka topic in spark. How can I start multiple consumers ? Do I need to have multiple instances of same application running using same group id ? or is there any configuration I can make use of while starting application and it does that job internally?

Answer 1

Passing --num-executors and using more than one core per executor will make more than one consumer thread in Spark

Each consumer thread gets mapped to a single partition.

Make the total threads equal the total partitions to maximize distributed throughput