Spark Streaming空RDD问题

Question

I am trying to create custom stream receiver from RDBMS.

val dataDStream = ssc.receiverStream(new inputReceiver ())
  dataDStream.foreachRDD((rdd:RDD[String],time:Time)=> {
    val newdata=rdd.flatMap(x=>x.split(","))
    newdata.foreach(println)  // *******This line has problem, newdata has no records
  })

ssc.start()
ssc.awaitTermination()
}

class inputReceiver extends Receiver[String](StorageLevel.MEMORY_AND_DISK_2) with Logging {
  def onStart() {
    // Start the thread that receives data over a connection
    new Thread("RDBMS data Receiver") {
      override def run() {
        receive()
      }
    }.start()
  }
  def onStop() {
  }

  def receive() {
    val sqlcontext = SQLContextSingleton.getInstance()

    // **** I am assuming something wrong in following code
    val DF = sqlcontext.read.json("/home/cloudera/data/s.json")
    for (data <- rdd) {
      store(data.toString())
    }
    logInfo("Stopped receiving")
    restart("Trying to connect again")
  }
}

The code is executing without error, but is not printing any record from dataframe.

I am using Spark 1.6 and Scala

Answer 1

To make your code work you should change the following:

def receive() {
  val sqlcontext = SQLContextSingleton.getInstance()
  val DF = sqlcontext.read.json("/home/cloudera/data/s.json")

  // **** this:
  rdd.collect.foreach(data => store(data.toString()))

  logInfo("Stopped receiving")
  restart("Trying to connect again")
}

HOWEVER this is not advisable because all of the data in your json file will be processed by the driver, and the Receiver hasn't been given proper consideration to reliability.

I suspect that Spark Streaming isn't the right fit for your use-case. Reading between the lines, it seems either you are streaming and as such need a proper producer, or you are reading data dumped from RDBMS into json, in which case you don't need Spark Streaming.