How can I make the most efficient Kafka healtcheck control?

Question

I use spring kafka. I want to determine that Kafka is closed. Does the kafka have such implementation? I found various solution. But all solutions that I found I have to write scheduled jobs. Is there a method that shows kafka is closed without writing scheduled job?

Solutions that I can find :

This solution was reverted for the reason exposed in

https://github.com/spring-projects/spring-boot/issues/12225

https://github.com/spring-projects/spring-boot/issues/12222

https://github.com/spring-projects/spring-boot/pull/12200

So, I dont want to use this solution.

@Configuration
public class KafkaConfig {

@Autowired
private KafkaAdmin admin;

@Autowired
private MeterRegistry meterRegistry;

@Autowired
private Map<String, KafkaTemplate<?, ?>> kafkaTemplates;

@Bean
public AdminClient kafkaAdminClient() {
    return AdminClient.create(admin.getConfig());
}

@SuppressWarnings("deprecation") // Can be avoided by relying on Double.NaN for non doubles.
@PostConstruct
private void initMetrics() {
    final String kafkaPrefix = "kafka.";
    for (Entry<String, KafkaTemplate<?, ?>> templateEntry : kafkaTemplates.entrySet()) {
        final String name = templateEntry.getKey();
        final KafkaTemplate<?, ?> kafkaTemplate = templateEntry.getValue();
        for (Metric metric : kafkaTemplate.metrics().values()) {
            final MetricName metricName = metric.metricName();
            final Builder<Metric> gaugeBuilder = Gauge
                    .builder(kafkaPrefix + metricName.name(), metric, Metric::value) // <-- Here
                    .description(metricName.description());
            for (Entry<String, String> tagEntry : metricName.tags().entrySet()) {
                gaugeBuilder.tag(kafkaPrefix + tagEntry.getKey(), tagEntry.getValue());
            }
            gaugeBuilder.tag("bean", name);
            gaugeBuilder.register(meterRegistry);
        }
    }
}

@Bean
public HealthIndicator kafkaHealthIndicator() {
    final DescribeClusterOptions describeClusterOptions = new DescribeClusterOptions().timeoutMs(1000);
    final AdminClient adminClient = kafkaAdminClient();
    return () -> {
        final DescribeClusterResult describeCluster = adminClient.describeCluster(describeClusterOptions);
        try {
            final String clusterId = describeCluster.clusterId().get();
            final int nodeCount = describeCluster.nodes().get().size();
            return Health.up()
                    .withDetail("clusterId", clusterId)
                    .withDetail("nodeCount", nodeCount)
                    .build();
        } catch (InterruptedException | ExecutionException e) {
            return Health.down()
                    .withException(e)
                    .build();
        }
    };

}
}

Another solution that I found:

public class KafkaHealthIndicator implements HealthIndicator {
private final Logger log = LoggerFactory.getLogger(KafkaHealthIndicator.class);

private KafkaTemplate<String, String> kafka;

public KafkaHealthIndicator(KafkaTemplate<String, String> kafka) {
    this.kafka = kafka;
}

/**
 * Return an indication of health.
 *
 * @return the health for
 */
@Override
public Health health() {
    try {
        kafka.send("kafka-health-indicator", "❥").get(100, TimeUnit.MILLISECONDS);
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
        return Health.down(e).build();
    }
    return Health.up().build();
}
}

In two solutions, I have to use scheduled job. Is there a method that shows kafka is closed without writing scheduled job? If no methods available, second solutions is enough to shows kafka is closed?

Answer 1

AFAIK, all Spring health checks are scheduled checks.

I've trusted the first code block you've put in other projects since it relies on a healthy Kafka Controller to return back a list of active brokers. One addition I would suggest is adding a field to the class for the minimum amount of brokers you expect to be part of the nodeCount