一个Spring的KafkaConsumer侦听器可以侦听来自同一分区的多个消息吗？

Question

Provided a topic having n partitions. Is there any way by which Spring's KafkaConsumer listener listens to multiple messages from same/one partition in one go?

I tried ConcurrentKafkaListenerContainerFactory by setting setBatchListener(true); but consumer has started consuming multiple messages from different partitions instead of one.

public class BatchReceiverConfig {

    @Value("${kafka.bootstrap-servers}")
    private String bootstrapServers;

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();

        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "batch4");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        // maximum records per batch receive
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "10");

        return props;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(2);
        // enable batch listeners
//        factory.setBatchListener(true);

        return factory;
    }

    @Bean
    public BatchReceiver receiver() {
        return new BatchReceiver();
    }
}


/* Listener */
@KafkaListener(id = "batch-listener", topics = TOPIC_TEST_BATCH)
    public void receive(List<String> data,
                        @Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
                        @Header(KafkaHeaders.OFFSET) List<Long> offsets) {

        LOGGER.info("start of batch receive");
        for (int i = 0; i < data.size(); i++) {
            LOGGER.info("received message='{}' with partition-offset='{}'", data.get(i),
                    partitions.get(i) + "-" + offsets.get(i));
            // handle message

            latch.countDown();
        }
        LOGGER.info("end of batch receive");
    }

Answer 1

假设组中只有一个使用者，则可以将该ConcurrentKafkaListener concurrencyLevel设置为n 。

Answer 2

If you use group management, Kafka will assign partitions to your consumers. Each consumer (or thread when using the concurrency) will get 0, 1, or more partitions, depending on how many consumers and partitions there are.

If you want a particular consumer to get messages from only one partition, you have to assign the partitions yourself instead of letting Kafka do the assignment.

If you use the concurrent container; you would have to assign the same number of partitions as the concurrency, so each thread only gets one partition.