Spark Streaming MQTT

Question

I've been using spark to stream data from kafka and it's pretty easy.

I thought using the MQTT utils would also be easy, but it is not for some reason.

I'm trying to execute the following piece of code.

  val sparkConf = new SparkConf(true).setAppName("amqStream").setMaster("local")
  val ssc = new StreamingContext(sparkConf, Seconds(10))

  val actorSystem = ActorSystem()
  implicit val kafkaProducerActor = actorSystem.actorOf(Props[KafkaProducerActor])

  MQTTUtils.createStream(ssc, "tcp://localhost:1883", "AkkaTest")
    .foreachRDD { rdd =>
      println("got rdd: " + rdd.toString())
      rdd.foreach { msg =>
        println("got msg: " + msg)
      }
    }

  ssc.start()
  ssc.awaitTermination()

The weird thing is that spark logs the msg I sent in the console, but not my println.

It logs something like this:

19:38:18.803 [RecurringTimer - BlockGenerator] DEBUG oassreceiver.BlockGenerator - Last element in input-0-1435790298600 is SOME MESSAGE

Answer 1

foreach is a distributed action, so your println may be executing on the workers. If you want to see some of the messages printed out locally, you could use the built in print function on the DStream or instead of your foreachRDD collect (or take) some of the elements back to the driver and print them there. Hope that helps and best of luck with Spark Streaming :)

Answer 2

If you wish to just print incoming messages, try something like this instead of the for_each (translating from a working Python version, so do check for Scala typos):

val mqttStream = MQTTUtils.createStream(ssc, "tcp://localhost:1883", "AkkaTest")
mqttStream.print()