How to close Kafka streams application when topic not available?

Question

I am using Kstreams with SpringBoot Application. I have added given below code to handle shutdown for the streams.

    KafkaStreams streams = new KafkaStreams(builder.build(), props);
    final CountDownLatch latch = new CountDownLatch(1);
    Runtime.getRuntime().addShutdownHook(new Thread("streams-shutdown-hook") {
        @Override
        public void run() {
            streams.close(Duration.ofMillis(30000));
            latch.countDown();
        }
    });
    try {
        streams.setUncaughtExceptionHandler((Thread t, Throwable e) -> {
            logger.error("Stream stopped with uncaught error", e);
            if (e.getCause() instanceof OffsetOutOfRangeException)
                streams.cleanUp();
            System.exit(1);
        });
        streams.start();
        latch.await();
    } catch (Throwable e) {
        logger.error("Stream stopped with error");
        System.exit(1);
    }

However, when a used topic is not available in kafka, the shutdown hook is never reached, the application just logs Shutdown complete and latch is stuck on await so the application never quits. How can I do a graceful shutdown in this case?

Answer 1

You can register a state listener ("state" means RUNNING, REBALANCING etc -- it's not related to state store...) on the KafkaStreams instance.

The state listener is invoked each time that client state changes. For the case you mention, the state would transit, too, so you can make sure to "unblock" the latch.

Cf. https://docs.confluent.io/current/streams/monitoring.html#status-of-kafkastreams-instances

Answer 2

Kafka consumers continuously look for their subscribed topics, even if they don't exist. This is more apparent when using regex patterns for subscription. UNKNOWN_TOPIC_OR_PARTITION is not an error condition in consumers.

You'd have to manually list the topics first, then check if that list contains the topic you're trying to read.