Spark streaming using MQTTutils to subscribe topic from activemq with authentication

Question

It seems MQTTUtils Only provide three methods, def createStream(jssc: JavaStreamingContext, brokerUrl: String, topic: String, storageLevel: StorageLevel): JavaDStream[String]

Create an input stream that receives messages pushed by a MQTT publisher. def createStream(jssc: JavaStreamingContext, brokerUrl: String, topic: String): JavaDStream[String]

Create an input stream that receives messages pushed by a MQTT publisher. def createStream(ssc: StreamingContext, brokerUrl: String, topic: String, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): DStream[String]

Create an input stream that receives messages pushed by a MQTT publisher.

But How can I provide username and password if the broker enabled authentication?

Answer 1

You could try including the username and password in the url:

mqtt://username:password@host:port

Answer 2

Please find this MQTT Scala Word Count Example .

Particular for your case run the publisher as

bin/run-example org.apache.spark.examples.streaming.MQTTPublisher mqtt://username:password@host:port foo

And Subscriber as

bin/run-example org.apache.spark.examples.streaming.MQTTWordCount mqtt://username:password@host:port foo

Before doing this ensure that you are started ActiveMQ broker.

example code

import org.apache.activemq.broker.{TransportConnector, BrokerService}
.
.
.
.
def startActiveMQMQTTBroker() {
    broker = new BrokerService()
    broker.setDataDirectoryFile(Utils.createTempDir())
    connector = new TransportConnector()
    connector.setName("mqtt")
    connector.setUri(new URI("mqtt:" + brokerUri))
    broker.addConnector(connector)
    broker.start()
}

pom file

<dependency>
  <groupId>org.apache.activemq</groupId>
  <artifactId>activemq-core</artifactId>
  <version>5.7.0</version>
</dependency>

Answer 3

You can trying using the customized spark-streaming-mqtt-connector library available here - https://github.com/sathipal/spark-streaming-mqtt-with-security_2.10-1.3.0 .

This library adds the following on top of the original library,

• Added TLS v1.2 security such that the communication is always secured.
• Stored topic along with the payload in the RDD.

So, use the following method to create the stream,

val lines = MQTTUtils.createStream(ssc, // Spark Streaming Context
            "ssl://URL",                // Broker URL
            "<topic>",                 // MQTT topic
            "MQTT client-ID",          // Unique ID of the application
            "Username", 
            "passowrd")

There are overloaded constructors that allows you to pass the RDD storage level as well. Hope this helps.