Kafka AdminClient 性能注意事项

Question

I would like to dynamically create Kafka topics. In my case, there can be up to several hundred topics in the application. There can be multiple concurrent calls to this method for each topic during system startup.

The AdminClient object has local scope, so it will be created every time. I suspect that a socket and a connection to the Kafka broker are opened underneath, so this solution is not optimal in terms of performance, as there may be several hundred connections open in memory at any one time.

import java.util.Collections;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import lombok.RequiredArgsConstructor;
import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.CreateTopicsResult;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.common.KafkaFuture;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
@RequiredArgsConstructor
class TopicFactory {

   private final Logger log = LoggerFactory.getLogger(TopicFactory.class);

   private final Set<String> topics = ConcurrentHashMap.newKeySet();

   @Value("${kafka.bootstrap.servers}")
   private final String bootstrapServers;

   @Value("${kafka.topic.replication.factor}")
   private final String replicationFactor;
   
   void createTopicIfNotExists(String topicName, int partitionCount) {
      if (topics.contains(topicName)) {
         return;
      }
      Properties properties = new Properties();
      properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
      try (Admin admin = Admin.create(properties)) {
         if (isTopicExists(admin, topicName)) {
            topics.add(topicName);
            return;
         }
         NewTopic newTopic = new NewTopic(topicName, partitionCount, Short.parseShort(replicationFactor));
         CreateTopicsResult result = admin.createTopics(Collections.singleton(newTopic));
         KafkaFuture<Void> future = result.values().get(topicName);
         try {
            future.get();
            topics.add(topicName);
         } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            log.error("Interrupted exception occurred during topic creation", e);
         } catch (ExecutionException e) {
            log.error("Execution exception occurred during topic creation", e);
         }
      }
   }

   private boolean isTopicExists(Admin admin, String topicName) {
      try {
         return admin.listTopics().names().get().contains(topicName);
      } catch (InterruptedException e) {
         Thread.currentThread().interrupt();
         log.error("Interrupted exception occurred during topic creation", e);
         return false;
      } catch (ExecutionException e) {
         log.error("Execution exception occurred during topic creation", e);
         return false;
      }
   }
}

How to improve the performance of this solution? Is connection caching a good idea? If so, in what way? As an initialized field in a class or maybe using eg Guava cache or Suppliers.memoize(...) ? However, then the connection with the broker would have to be maintained all the time.

Answer 1

If you want to improve this solution for hundreds of topics, as it is written, then admin.createTopics takes a whole collection, so don't use a singleton list.

Also, admin.listTopics() result can be cached so that you don't query all topics every time you create one more topic.

Otherwise, I personally would use alternative solutions like Terraform rather than Spring. Since topics aren't going to need to be recreated very often (in the same Kafka cluster, at least), so your code might only be ran a handful of times, but you're needlessly increasing the size of your Spring app by dragging that TopicFactory class around.