using mqtt protocol with kafka as a message broker

Question

How can we use mqtt protocol with kafka as a message broker?

The clients(android/ios/desktop java apps etc) will be producing and consuming messages using mqtt phao client side libraries which are available in different languages using kafka as a message broker.

Any advice?

Answer 1

You can use a Kafka source connector which will stream data from an MQTT broker like Mosquitto into a Kafka cluster. See https://github.com/evokly/kafka-connect-mqtt

The simplest way to run the connector is in standalone mode, where a single instance will be running on the Kafka cluster on a single node. You can also run it in distributed mode (albeit with much more configuration) and this will distribute the connector across the cluster for greater throughput. In the distributed mode you can devise a topology that allows horizontal scaling, parallel throughput and high availability. Implementing additional guarantees requires additional load balancers, multiple MQTT brokers and last will and testament scenarios to deal with connectors crashing, but this is probably out of scope for this question.

Using the connector approach has the advantage of the Kafka cluster making sure your connector is alive and re-starting it if necessary. Distributed mode offers even more advantages.

Answer 2

AFAIK, there's no "official" MQTT connector for Kafka. But you can use MQTT-Kafka bridge. For inspiration look at https://github.com/km4rcus/mqttKafkaBridge (Please note there's a bug in this implementation: kafka topics can't contain "/" so you will probably want to replace them with "_" in messageArrived in Bridge.java file)

Note that this code is just a very simple solution, not scalable. It's a good idea probably to write your custom implementation to suit your expectations better. But you should keep it as simple as possible - it's a single point of failure. As long as you get your data into Kafka, you get some guarantees but you get no guarantees from MQTT broker. When the bridge crashes, you are simply loosing your data...

Answer 3

This is not a good idea. A MQTT client usually is very light-weight with limited resources. Those devices or IoT has small memory/CPU power. A Kafka client generally is heavy-weight. For example, Kafka client has to keep track of the offset. It also requires interaction to Zookeepers. In short, Kafka is not suitable as MQTT broker. You are better off choosing a popular MQTT broker such as Mosquito.

Answer 4

As mentioned by @Miroslav Prymek there is no official bridge. The one has been created by Jacklund. I have upgraded the bridge to support Kafka v 0.8.1.1. Also contains a prebuilt binary compatible with JRE 1.7. Here is the link to the MQTT Kafka Bridge .

Answer 5

You could use Mosca , an mqtt adapter written in javascript.

Mosca is a node.js mqtt broker, which can be used standalone or embedded in another Node.js application

Mosca supports different backends such as redis and mongodb, but also kafka. A Kafka MQTT Bridge application is included in the Mosca examples .

This article on linkedin describes the solution more fully.

Disclaimer: I am a contributor to the Mosca Kafka Bridge example application.