Springboot KafkaListener 監聽多個主題

Question

我正在嘗試構建一個 kafka 服務，其中有兩個主題和一個消費者。 每個主題有 10 個分區。 目前我有一個設置，消費者可以同時收聽這兩個主題。

我想讓處理優先級為基礎。 我想優先考慮主題 1 中的事件而不是主題 2。

所以目標是，僅當主題 1 中沒有事件時才處理主題 2 中的事件。我一直在嘗試 Consumer.Pause 選項，但到目前為止沒有運氣，因為我是 Kafka 的初學者。

欣賞一個代碼片段，它可以顯示我需要什么來實現這一目標..

Answer 1

這是一種解決方案：

@SpringBootApplication
public class So63855196Application {

    public static void main(String[] args) {
        SpringApplication.run(So63855196Application.class, args);
    }

    @Bean
    public NewTopic hiT() {
        return TopicBuilder.name("hiT").partitions(1).replicas(1).build();
    }

    @Bean
    public NewTopic loT() {
        return TopicBuilder.name("loT").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> template.send("loT", "foo" + i));
            Thread.sleep(3000);
            IntStream.range(0,  2).forEach(i -> template.send("hiT", "bar" + i));
        };
    }

}

@Component
class Listener {

    private static final Logger log = LoggerFactory.getLogger(Listener.class);

    private final KafkaListenerEndpointRegistry registry;

    private final AtomicBoolean paused = new AtomicBoolean();

    Listener(KafkaListenerEndpointRegistry registry) {
        this.registry = registry;
    }

    @KafkaListener(id = "high", topics = "hiT")
    @KafkaListener(id = "low", topics = "loT",
            properties = ConsumerConfig.MAX_POLL_RECORDS_CONFIG + ":1")
    public void listen(String in, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) throws InterruptedException {
        System.out.println(in + " from " + topic);
        if (topic.equals("hiT")) {
            if (this.paused.compareAndSet(false, true)) {
                log.info("Hi-pri active; pausing low-pri");
                this.registry.getListenerContainer("low").pause();
            }
        }
        Thread.sleep(2000);
    }

    @EventListener
    public void event(ListenerContainerIdleEvent event) {
        if (event.getListenerId().startsWith("high")) {
            if (this.paused.compareAndSet(true, false)) {
                log.info("Hi-pri idle; resuming low-pri");
                this.registry.getListenerContainer("low").resume();
            }
        }
    }

}

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.idle-event-interval=1000

和

foo0 from loT
foo1 from loT
bar0 from hiT
2020-09-14 09:40:28.630  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri active; pausing low-pri
bar1 from hiT
2020-09-14 09:40:37.647  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri idle; resuming low-pri
foo2 from loT
foo3 from loT
foo4 from loT
foo5 from loT
foo6 from loT
foo7 from loT
foo8 from loT
foo9 from loT

這僅在容器具有concurrency=1時才有效； 當容器有多個線程時，它需要更復雜一些。