[英]How can I consume from multiple Kafka topics that are associated with different brokers?
How can I consume from multiple Kafka topics that are associated with different brokers?如何从与不同代理关联的多个 Kafka 主题中消费?
I have a Spring Boot application that needs to consume from 2 topics, but the topics are associated with different Brokers.我有一个 Spring 启动应用程序需要从 2 个主题中消费,但这些主题与不同的经纪人相关联。
I am using Spring Kafka with the @Listener annotation, and I see there are ways to consume from 2 topics that are associated with the same broker, not different brokers.我正在使用带有 @Listener 注释的 Spring Kafka,我看到有一些方法可以从与同一代理相关联的 2 个主题中消费,而不是不同的代理。 Unfortunately, I don't see anything helpful in the Spring Boot or Spring Kafka docs on how to do this.不幸的是,我在 Spring Boot 或 Spring Kafka 文档中没有看到任何有关如何执行此操作的有用信息。
There are a few ways to do this, and unfortunately Spring-Boot and Spring-Kafka do not make it clear the best practice to implement this.有几种方法可以做到这一点,不幸的是 Spring-Boot 和 Spring-Kafka 并没有明确实现这一点的最佳实践。 There are also lots of answers across SO that address consuming from multiple topics with the same Broker, and it isn't always as simple as that. SO中还有很多答案可以解决使用同一个Broker从多个主题中消费的问题,但这并不总是那么简单。
The easiest way to solve this is to add a properties parameter in your Kafka Listener annotations :解决此问题的最简单方法是在Kafka 侦听器注释中添加属性参数:
@KafkaListener(topics = ["\${topic-1-name}"], properties = ["bootstrap.servers=\${bootstrap-server-1}"])
fun topic1Listener(@Payload messages: List<String>, ack: Acknowledgment){
// Do work
}
@KafkaListener(topics = ["\${topic-2-name}"], properties = ["bootstrap.servers=\${bootstrap-server-2}"])
fun topic2Listener(@Payload messages: List<String>, ack: Acknowledgment){
// Do work
}
Whatever key/value pair we specify in the properties param value will override the default key/value in the DefaultKafkaConsumerFactory.我们在属性参数值中指定的任何键/值对都将覆盖 DefaultKafkaConsumerFactory 中的默认键/值。 In this case, we override the bootstrap.servers property to our own specific bootstrap server address for each topic.在这种情况下,我们将 bootstrap.servers 属性覆盖为我们自己的每个主题的特定引导服务器地址。
However, we can still use the Spring-Boot features that are “nice to have”, such as auto-topic creation and allowing Spring-Boot to setup a group-id for our application.但是,我们仍然可以使用 Spring-Boot 的“不错的”功能,例如自动主题创建和允许 Spring-Boot 为我们的应用程序设置组 ID。 We just need to leave our group-id parameter in our application.properties or application.yml file.我们只需要将 group-id 参数保留在 application.properties 或 application.yml 文件中。
spring:
kafka:
consumer:
group-id: group-id-of-your-choice
There are lots of other ways to accomplish this, however this is the simplest way I have found and tested to work.还有很多其他方法可以实现这一点,但这是我发现并测试过的最简单的方法。
Other methods include creating your own custom ConsumerFactory and KafkaListenerContainerFactory objects, then configuring the properties in each Factory to use the bootstrap servers of your choice.其他方法包括创建您自己的自定义 ConsumerFactory 和 KafkaListenerContainerFactory 对象,然后配置每个 Factory 中的属性以使用您选择的引导服务器。 However, the 1st method is much cleaner and simpler, using the default Container Factory.但是,第一种方法更简洁,使用默认的 Container Factory。 Below is how to create custom Factories with your own properties.以下是如何使用您自己的属性创建自定义工厂。
@Bean
fun ConsumerFactory1(): DefaultKafkaConsumerFactory<String, String> {
val props = mutableMapOf<String, Any>()
props[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] = bootStrapServers1!!
props[ConsumerConfig.GROUP_ID_CONFIG] = groupId!!
props[ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
props[ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
return DefaultKafkaConsumerFactory(props)
}
@Bean
fun ContainerFactory1(): ConcurrentKafkaListenerContainerFactory<String, String>? {
val factory: ConcurrentKafkaListenerContainerFactory<String, String> = ConcurrentKafkaListenerContainerFactory()
factory.consumerFactory = ConsumerFactory1()
factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL
factory.isBatchListener = true
return factory
}
@Bean
fun ConsumerFactory2(): DefaultKafkaConsumerFactory<Any?, Any?> {
val props = mutableMapOf<String, Any>()
props[ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG] = bootStrapServers2!!
props[ConsumerConfig.GROUP_ID_CONFIG] = groupId!!
props[ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
props[ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java
}
@Bean
fun ContainerFactory2(): ConcurrentKafkaListenerContainerFactory<String, String> {
val factory: ConcurrentKafkaListenerContainerFactory<String, String> = ConcurrentKafkaListenerContainerFactory()
factory.consumerFactory = ConsumerFactory2()
factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL
factory.isBatchListener = true
return factory
}
A few things to unpack here.这里有一些东西要解压。
@KafkaListener(topics = ["\${kafka-topic-1}"], containerFactory = "ContainerFactory1", groupId = "\${kafka.group-id}")
There are even more ways to accomplish implementing this, and I know others have come up with good solutions too, sometimes it all depends on what you need for your application.还有更多方法可以实现这一点,我知道其他人也提出了很好的解决方案,有时这完全取决于您的应用程序需要什么。 This is just 2 of the solutions I have found to be successful with, and Method 1 is easy to understand, implement and test without getting too involved with the depth of Spring-Boot and Spring-Kafka.这只是我发现成功的解决方案中的 2 个,方法 1 易于理解、实施和测试,无需过多涉及 Spring-Boot 和 Spring-Kafka 的深度。 These methods will work with more than just 2 brokers if that is functionality you need.如果这是您需要的功能,这些方法将与超过 2 个代理一起使用。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.