Provided a topic having n partitions. Is there any way by which Spring's KafkaConsumer listener listens to multiple messages from same/one partition in one go?
I tried ConcurrentKafkaListenerContainerFactory
by setting setBatchListener(true);
but consumer has started consuming multiple messages from different partitions instead of one.
public class BatchReceiverConfig {
@Value("${kafka.bootstrap-servers}")
private String bootstrapServers;
@Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "batch4");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// maximum records per batch receive
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "10");
return props;
}
@Bean
public ConsumerFactory<String, String> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(2);
// enable batch listeners
// factory.setBatchListener(true);
return factory;
}
@Bean
public BatchReceiver receiver() {
return new BatchReceiver();
}
}
/* Listener */
@KafkaListener(id = "batch-listener", topics = TOPIC_TEST_BATCH)
public void receive(List<String> data,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
@Header(KafkaHeaders.OFFSET) List<Long> offsets) {
LOGGER.info("start of batch receive");
for (int i = 0; i < data.size(); i++) {
LOGGER.info("received message='{}' with partition-offset='{}'", data.get(i),
partitions.get(i) + "-" + offsets.get(i));
// handle message
latch.countDown();
}
LOGGER.info("end of batch receive");
}
假设组中只有一个使用者,则可以将该ConcurrentKafkaListener
concurrencyLevel
设置为n
。
If you use group management, Kafka will assign partitions to your consumers. Each consumer (or thread when using the concurrency) will get 0, 1, or more partitions, depending on how many consumers and partitions there are.
If you want a particular consumer to get messages from only one partition, you have to assign the partitions yourself instead of letting Kafka do the assignment.
If you use the concurrent container; you would have to assign the same number of partitions as the concurrency, so each thread only gets one partition.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.