在使用RDD的Scala中，如果RDD [（k，Iterable [v]），如何在Iterable中获得Apply函数

Question

I am trying to figure a way to efficiently apply filters (multiple) on the Iterable[Value] for a given RDD[Key,Iterable[Value]] class.

The reason being is I want to filter the RDD and eventually find the keys that match the filter

Example of RDD

 000473643-02,CompactBuffer((glucose,80.0), (glucose,80.0), (glucose2,80.0),   (fasting blood glucose,80.0), (glucose,80.0), (glucose,80.0), (glucose,80.0), (glucose,80.0)))
 (713003448-01,CompactBuffer((glucose,80.0), (glucose,80.0)))
 (000023838-01,CompactBuffer((glucose,80.0), (glucose,80.0)))
 (000772974-01,CompactBuffer((glucose,80.0), (glucose,80.0), (glucose,80.0)))
 (380670000-01,CompactBuffer((glucose,80.0), (glucose,80.0)))

So in this case, I need to output the Key only when the the following is true:

    glucose value is >= 80 or fasting blood glucose >= 80 

Answer 1

I would use something like this:

case class ExceedsCondition(threshold:Double) {
  def violates(value:Double) = value >= threshold
}

val conditionsBroadcast = sc.broadcast(
  Map("glucose" -> ExceedsCondition(80.0), 
      "fasting-glucose" -> ExceedsCondition(81.0))
  )

val rdd = sc.parallelize(List("key1" -> List(("glucose" -> 83.0))))

val result = rdd.filter { case (_, xs) =>
            val conditions = conditionsBroadcast.value
            xs.exists { case (key, value) =>
                conditions.get(key).exists(_.violates(value))
            }
}

result.take(10)