Springboot KafkaListener 监听多个主题

Question

我正在尝试构建一个 kafka 服务，其中有两个主题和一个消费者。 每个主题有 10 个分区。 目前我有一个设置，消费者可以同时收听这两个主题。

我想让处理优先级为基础。 我想优先考虑主题 1 中的事件而不是主题 2。

所以目标是，仅当主题 1 中没有事件时才处理主题 2 中的事件。我一直在尝试 Consumer.Pause 选项，但到目前为止没有运气，因为我是 Kafka 的初学者。

欣赏一个代码片段，它可以显示我需要什么来实现这一目标..

Answer 1

这是一种解决方案：

@SpringBootApplication
public class So63855196Application {

    public static void main(String[] args) {
        SpringApplication.run(So63855196Application.class, args);
    }

    @Bean
    public NewTopic hiT() {
        return TopicBuilder.name("hiT").partitions(1).replicas(1).build();
    }

    @Bean
    public NewTopic loT() {
        return TopicBuilder.name("loT").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> template.send("loT", "foo" + i));
            Thread.sleep(3000);
            IntStream.range(0,  2).forEach(i -> template.send("hiT", "bar" + i));
        };
    }

}

@Component
class Listener {

    private static final Logger log = LoggerFactory.getLogger(Listener.class);

    private final KafkaListenerEndpointRegistry registry;

    private final AtomicBoolean paused = new AtomicBoolean();

    Listener(KafkaListenerEndpointRegistry registry) {
        this.registry = registry;
    }

    @KafkaListener(id = "high", topics = "hiT")
    @KafkaListener(id = "low", topics = "loT",
            properties = ConsumerConfig.MAX_POLL_RECORDS_CONFIG + ":1")
    public void listen(String in, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) throws InterruptedException {
        System.out.println(in + " from " + topic);
        if (topic.equals("hiT")) {
            if (this.paused.compareAndSet(false, true)) {
                log.info("Hi-pri active; pausing low-pri");
                this.registry.getListenerContainer("low").pause();
            }
        }
        Thread.sleep(2000);
    }

    @EventListener
    public void event(ListenerContainerIdleEvent event) {
        if (event.getListenerId().startsWith("high")) {
            if (this.paused.compareAndSet(true, false)) {
                log.info("Hi-pri idle; resuming low-pri");
                this.registry.getListenerContainer("low").resume();
            }
        }
    }

}

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.idle-event-interval=1000

和

foo0 from loT
foo1 from loT
bar0 from hiT
2020-09-14 09:40:28.630  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri active; pausing low-pri
bar1 from hiT
2020-09-14 09:40:37.647  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri idle; resuming low-pri
foo2 from loT
foo3 from loT
foo4 from loT
foo5 from loT
foo6 from loT
foo7 from loT
foo8 from loT
foo9 from loT

这仅在容器具有concurrency=1时才有效； 当容器有多个线程时，它需要更复杂一些。