Kafka topics as chatroom identifiers

Question

I am interested in a messaging chat app where different chatroom can be possible. I try to understand if Kafka topics can be used as chatroom identifiers from Kafka conceptual point of view?

As chatrooms are normally created in a runtime then it means that

• Kafka topics should be created in a runtime. Is it possible?
• Users should be able to subscribe to the new topics (chatrooms) as soon as they are created. Is it possible?
• There will be a lot of chatrooms. Is it bad or ok for Kafka to have huge number of topics?

Answer 1

Kafka topics should be created in a runtime. Is it possible?

Yes. Kafka AdminClient provides a programmatic API for administrative functionality, including listing, creating and deleting topics.

We can use createTopics to create new topics:

Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
AdminClient admin = AdminClient.create(props);


CreateTopicsResult newTopic = admin.createTopics(Collection.singletonList(new NewTopic(TOPIC_NAME, NUM_PARTITIONS, REP_FACTOR)));

admin.close(Duration.ofSeconds(30));

The above code comes from Kafka: The Definitive Guide 2nd Edition, Chapter 5 Managing Apache Kafka Programmatically, which discusses a lot about AdminClient.

For spring-kafka, the KafkaAdmin provides createOrModifyTopics() method.

Users should be able to subscribe to the new topics (chatrooms) as soon as they are created. Is it possible?

Yes. We can use regular expressions in .subscribe() :

The expression can match multiple topic names, and if someone creates a new topic with a name that matches, a rebalance will happen almost immediately and the consumers will start consuming from the new topic.

For example, to subscribe to all test topics, we can call

consumer.subscribe(Pattern.compile("test.*"));

There will be a lot of chatrooms. Is it bad or ok for Kafka to have huge number of topics?

It's okay: Can I have 100s of thousands of topics in a Kafka Cluster?