Kafka Streams Shutdown Hook and Unexpected Exception Handling in the same Stream application

Question

I was tasked with tearing down a Dev environment and setting it up again from scrap to verify our CI-CD processes; only problem was that I messed up creating one topic and so the Kafka Streams application exited with an error.

I dug into it and found the issue and corrected it but as I was digging in I ran into another odd little problem.

I implemented a Unexpected Exception handler as so:

streams.setUncaughtExceptionHandler((t, e) -> {
    logger.fatal("Caught unhandled Kafka Streams Exception:", e);
    // Do some exception handling.
    streams.close();

    // Maybe do some more exception handling.
    // Open a lock that is waiting after streams.start() call 
    // to let application exit normally
    shutdownLatch.countDown();
});

Problem is that if the application threw an exception because of a topic error when the KafkaStreams::close is call the application seems to dead lock in WindowsSelectorImpl::poll after attempting a call to KafkaStreams::waitOnState.

I thought it might be an issue with calling KafkaStreams::close inside the Exception Handler but I found this SO and a comment from Matthias J. Sax that said it should be ok to call KafkaStreams::Close in the exception handler with the caveat of not calling KafkaStreams::close from multiple threads.

The issue is that I want to implement a shutdown hook to kill the steams application gracefully on request, as well as implement the UnexpectedException handler to clean up and terminate gracefully in the event of exceptions.

I came up with the following solution that checks the KafkaStreams state before calling close and it does actually work, but it seems a bit iffy since I could see other cases besides running (perhaps Pending) where we would want to ensure the KafkaStreams::close it called.

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    logger.fatal("Caught Shutdown request");
    // Do some shutdown cleanup.
    if (streams.state().isRunning())
    {
        If this hook is called due to the Main exiting after handling 
        an exception we don't want to call close again. It doesn't 
        cause any errors but logs that the application was closed 
        a second time.
        streams.close(100L, TimeUnit.MILLISECONDS);
    }
    // Maybe do a little bit more clean up before system exits.
    System.exit(0);

}));

streams.setUncaughtExceptionHandler((t, e) -> {
    logger.fatal("Caught unhandled Kafka Streams Exception:", e);
    // Do some exception handling.
    if (streams.state().isRunning())
    {
        streams.close(100L, TimeUnit.MILLISECONDS);
    }
    // Maybe do some more exception handling.

    // Open the Gate to let application exit normally
    shutdownLatch.countDown();
    // Or Optionally call halt to immediately terminate and prevent call to Shutdown hook.
    Runtime.getRuntime().halt(0);
});

Any suggestions about why calling the KafkaSteams:close in the exception handler would be causing such troubles or if there is a better way to implement shutdown hook and the exception handler at the same time it would be greatly appreciated?

Answer 1

Calling close() from an exception handler and from shutdown hook is slightly different. close() might deadlock if called from the shutdown hook (cf. https://issues.apache.org/jira/browse/KAFKA-4366 ) and thus, you should call it with a timeout.

Also, the issue is related with calling System.exit() from within an uncaught exception handler as described in the Jira. In general, calling System.exit() is quite harsh and should be avoided IMHO.

Your solution seems not to be 100% robust, either, because streams.state().isRunning() could result in a race condition.

An alternative to using a timeout might be to only set an AtomicBoolean within both the shutdown hook and the exception handler and use the "main()" thread to call close if the boolean flag is set to true:

private final static AtomicBoolean stopStreams = new AtomicBoolean(false);

public static void main(String[] args) {
  // do stuff

  KafkaStreams streams = ...
  stream.setUncaughtExceptionHandler((t, e) -> {
    stopStreams.set(true);
  });

  Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    stopStreams.set(true);
  });

  while (!stopStreams.get()) {
    Thread.sleep(1000);
  }
  streams.close();
}