IoT - multiple Kafka producers to publish messages to same topic

Question

I am trying to develop an IoT based application, where multiple devices will generate the data and send it to a Kafka broker. The possible count of the devices would be around 60-70 thousand.

There could be 5-10 different device types, however, the number of device count for each type would be in thousand such as 20-40 thousand each.

I want to understand the Kafka topic usage in which it should support these many devices (20-40 thousand devices on average).

Also, please let me know whether the MQTT based implementation is required for such kind of applications.

Thanks in advance,

Avinash Deshmukh

Answer 1

The number of partitions mostly influence the consumer side, because it's the unit of parallelism for reading messages. On the producer side, consider that because each leader partition is hosted by a broker, the producer has to connect to different brokers for writing to different partitions so multiple TCP connections would be needed.

Regarding MQTT, it means that you have to run an MQTT broker and then Kafka Connect with the related MQTT connector. It would make sense depending on the kind of devices you have. MQTT protocol is most suitable for embedded/IoT devices as lightweight protocol, so for low power devices it could make more sense than having a Kafka protocol on the device stack. But it could also depend if you are using an IoT gateway at the edge gathering messages from multiple devices on the field and then sending to Kafka.

Answer 2

I want to understand the Kafka topic usage in which it should support these many devices (20-40 thousand devices on average).

To be honest I don't understand the question. Are you asking for topic sizing? number of partitions? etc etc. Are you going to provide your devices direct access to the Kafka cluster?

Also, please let me know whether the MQTT based implementation is required for such kind of applications.

Apache Kafka doesn't support MQTT per-se. Are you talking about any kind of commercial solution for that?