I need to convert org.apache.spark.rdd.RDD[Array[Byte]] to Array[Byte] in Spark Streaming

Question

I am trying to parse json file into spark streaming through kafka.

I wrote a function which requires Array[Byte] .

stream.foreachRDD(rdd=>
    parseAVROToString(rdd)
)

ssc.start

To use parseAVROToString I need Array[Byte] . I am getting the following error message :

found   : org.apache.spark.rdd.RDD[Array[Byte]] required: Array[Byte] parseAVROToString(rdd)

Answer 1

You can try this:

stream.foreachRDD(rdd => {
  rdd.foreach(record = > parseAVROToString(record))
})

But foreachRDD is an output operation which used to to save underline rdds to external files or databases. I think stream.transform might be useful in your case.