Custom Receiver stalls worker in Spark Streaming

Question

I am trying to write a Spark Streaming Application with a Custom Receiver. I is supposed to simulate real-time input data by providing random values with a pre-defined interval. The (simplified) receiver looks as follows, with the according Spark Streaming app code below:

class SparkStreamingReceiver extends Actor with ActorHelper {

  private val random = new Random()

  override def preStart = {
    context.system.scheduler.schedule(500 milliseconds, 1000 milliseconds)({
        self ! ("string", random.nextGaussian())
    })
  }

  override def receive = {
    case data: (String, Double) => {
      store[(String, Double)](data)
    }
  }
}

val conf: SparkConf = new SparkConf()
conf.setAppName("Spark Streaming App")
    .setMaster("local")

val ssc: StreamingContext = new StreamingContext(conf, Seconds(2))

val randomValues: ReceiverInputDStream[(String, Double)] =
    ssc.actorStream[(String,Double)](Props(new SparkStreamingReceiver()), "Receiver")

randomValues.saveAsTextFiles("<<OUTPUT_PATH>>/randomValues")

Running this code, I see that the receiver is working ( Storing item , received single log entries). However, saveAsTextFiles will never output values.

I can work around the problem by changing the master to run with two threads ( local[2] ), but if I register another instance of my receiver (which I intend to do), it reappears. More specifically, I need to have at least one thread more than the number of my custom receivers registered to get any output.

It seems to me as though the worker threads are stalled by the receivers.

Can anyone explain this effect, and possibly how to fix my code?

Answer 1

Each receiver uses a compute slot. So 2 receivers will require 2 compute slots. If all the compute slots are taken by receivers, then there is no slot left to process the data. That is why "local" mode with 1 receiver, and "local[2]" with 2 receivers stalls the processing.