如何在 spark kafka 流中创建消费者组并将消费者分配给消费者组

Question

I have one topic having name topic_1 and created 4 partitions. I need to read parallel in Kafka spark stream. so I need to make one consumer group and consumers.

Can you plz help how can I do this?

For now Kafka spark stream, one time taking one request from Kafka.

Answer 1

Assuming you are using KafkaUtils from Spark, it automatically will take advantage of the number of Spark Executors * Cores per Executor.

So, if you have 2 Spark Executors, with 2 Cores for each Executor, Spark will automatically consume 4 topic partitions in parallel.

In Kafka Spark Streaming integration, the number of input tasks are determined by the number of partitions in the topic. If your topic has 4 partitions, Spark Streaming will spawn 4 tasks for each batch.

If you have 1 Executor with 1 Core, then the core will sequentially executes the 4 tasks (no paralellism). Whereas if you have 2 Executor with 1 Core each, then each core will sequentially executes 2 tasks (so parallelism is 2).

With 4 partitions you should configure any of the following, to achieve max consumer parallellism:

• 1 Executor with 4 Cores
• 2 Executor with 2 Cores each
• 4 Executor with 1 Core each