Increase number of topic in Kafka leads zookeeper fail

Question

We plan to use kafka as a message broker for IoT use case, where each device is considered as unique topic. when I simulated 10 message per second to 10 thousand topics zookeeper is getting bottle neck,all Kafka monitoring tools fails to read the throughput values and number of topics from JMX port because of that. will tuning zookeeper will solve the issues. where In IoT use case there will be millions of device polling data to millions of topics. I want to make sure the approach is perfect to go. Please suggest.

Answer 1

There is another solution available now - try out Apache Pulsar. It looks promising about the number of topics and generally quite similar to Kafka and it is now compatible with Kafka. https://pulsar.apache.org/

Answer 2

Your right in saying that even though theoretically you could have millions of topics in Kakfa, the number of topics in Kafka is bound by Zookeeper , You cannot have million of topics. Creating topics/partitions per device is not scalable solution.

Is there a reason why you need millions of topics ?