检测 Kafka 集群已关闭的最佳方法是什么？

Question

The Kafka Consumer API is so nice to hide any transient connection errors and just pick up reading from it's current offset if a Kafka broker dies and comes up again.

But in some applications it's important to alert and stop processing data (from other sources), if the entire Kafka cluster is down (ie all brokers). I've browsed the misc. APIs and that doesn't seem to be a feature.

The closest I've come is to submit an Admin call and depending on a timeout, conclude that the Kafka cluster is down:

Properties properties   = ... // Load properties from somewhere.
int timeout             = 5_000; // 5 second timeout
AdminClient adminClient = AdminClient.create(properties);
try {
    adminClient.listTopics(new ListTopicsOptions().timeoutMs(timeout)).listings().get();
    // Here we know the cluster is up as call returned within timeout.
} catch (ExecutionException ex) {
    // Here we know that the cluster is down as the call timed out.
}

Is this the best way to do it?

Another way is to query ZooKeeper, but the above approach will also work in situations where there's a network problem between the application and Kafka.

Answer 1

Your approach looks fine. A similar approach (using Spring's HealthIndicator 's notion) is what MartinX3 did here .

His solution:

@Component
public class KafkaHealthIndicator implements HealthIndicator {
    private final Logger log = LoggerFactory.getLogger(KafkaHealthIndicator.class);

    private KafkaTemplate<String, String> kafka;

    public KafkaHealthIndicator(KafkaTemplate<String, String> kafka) {
        this.kafka = kafka;
    }

    /**
     * Return an indication of health.
     *
     * @return the health for
     */
    @Override
    public Health health() {
        try {
            kafka.send("kafka-health-indicator", "❥").get(100, TimeUnit.MILLISECONDS);
        } catch (InterruptedException | ExecutionException | TimeoutException e) {
            return Health.down(e).build();
        }
        return Health.up().build();
    }
}

You might also want to combine other metric-checks , before returning Health.up().build() , such as ActiveControllerCount = 0 , depending on what you consider important for your use case to consider the entire cluster as down.

Answer 2

Although I would suggest you to use a proper monitoring tool, in case you still want to do this programmatically, one option is to use AdminClient and try fetching the topic names.

---

For example,

Properties properties = new Properties();
properties.put("bootstrap.servers", "localhost:9092");
properties.put("request.timeout.ms", 5000);

try {

    AdminClient adminClient = AdminClient.create(properties)

    ListTopicsResult topics = adminClient.listTopics();
    Set<String> names = topics.names().get();

} catch(InterruptedException | ExecutionException e) {
    System.err.println("Kafka is unavailable");
}

Note however that the above won't throw an exception if some of the brokers are down (but obviously, if a broker is down it doesn't mean that the Kafka Cluster itself is down, as data should still be accessible)