计算RDD中的平均值，然后根据Spark Streaming中的平均值过滤此RDD

Question

I want to do something i feel very strange in Spark Streaming and i want to get some feedback on it.

I have a DStream of a tuple (String,Int). Let's say the string is an id and the integer a value.

So for a microbatch, I want to compute The average of the field Int, and base on this average i want to filter the same microbatch, for example field2 > average. So i wrote this code :

lineStreams
  .foreachRDD(
    rdd => {
      val totalElement = rdd.count()
      if(totalElement > 0) {
        val totalSum = rdd.map(elem => elem.apply(1).toInt).reduce(_ + _)
        val average = totalSum / totalElement
        rdd.foreach(
          elem => {
            if(elem.apply(1).toInt > average){
              println("Element is higher than average")
            }
          }
        )
      }
    })

But actually this code is not running, the first part of computation looks ok but not the test. I know there is some dirty things in this code, but I just want to know if the logic is good.

Thanks for you advices !

Answer 1

Try:

lineStreams.transform { rdd => {
  val mean = rdd.values.map(_.toDouble).mean
  rdd.filter(_._2.toDouble > mean)
}}