如何编写一个接收映射函数到泛型类型的scala函数

Question

Using Spark 1.3.0 with Scala, I have two functions which basically do the same on a given RDD[(Long, String, Boolean, String)] , up to a specifc map function from (Long, String, Boolean, String) to a tuple of 2 elements:

def rddToMap1(rdd: RDD[(Long, String, Boolean, String)]): Map[Long, Set[(String, Boolean)]] = {
rdd
  .map(t => (t._1, (t._2, t._3))) //mapping function 1
  .groupBy(_._1)
  .mapValues(_.toSet)
  .collect
  .toMap
  .mapValues(_.map(_._2))
  .map(identity)
}


def rddToMap2(rdd: RDD[(Long, String, Boolean, String)]): Map[(Long, String), Set[String]] = {
rdd
  .map(t => ((t._1, t._2), t._4)) //mapping function 2
  .groupBy(_._1)
  .mapValues(_.toSet)
  .collect
  .toMap
  .mapValues(_.map(_._2))
  .map(identity)
}

I want to write a generic function genericRDDToMap which I would later use to implement rddToMap1 and rddToMap2 .

This doesn't work:

def genericRDDToMap[A](rdd: RDD[(Long, String, Boolean, String)], mapFn: (Long, String, Boolean, String) => A) = {      
rdd     
  .map(mapFn) //ERROR       
  .groupBy(_._1)        
  .mapValues(_.toSet)       
  .collect      
  .toMap        
  .mapValues(_.map(_._2))       
  .map(identity)        
}

The (Eclipse) interpreter doesn't take mapFn as a valid mapping function, it says:

type mismatch; found : (Long, String, Boolean, String) => A required: ((Long, String, Boolean, String)) => ?

And even if I got over this, how would it know that my generic type A has value _1 in the groupBy to follow?

To summarize: how do I do it right?

Answer 1

You missed parentheses around (Long, String, Boolean, String) . And if A is of type TupleX , you can use upper bound to specify it (here I used Tuple2 ):

  def genericRDDToMap[X, Y, A <: Tuple2[X,Y]](rdd: RDD[(Long, String, Boolean, String)], 
                         mapFn: ((Long, String, Boolean, String)) => A) (implicit ev: ClassTag[A])= {     
      ... 
  }