获取Scala / Spark中的RDD类型

Question

I am not sure if type is the right word to use here, but let say I have an RDD of the following type

RDD[(Long, Array[(Long, Double)])]

Now if I have the RDD, how can i find the type of it (as mentioned above) at runtime ?

I basically want to compare two RDDs, at runtime to see if they store the same kind of data (the values it self might be different), is there another way to do it? Moreover, I want to get a cached RDD as an instance of RDD type using the following code

sc.getPersistentRDDs(0).asInstanceOf[RDD[(Long, Array[(Long, Double)])]]

where RDD[(Long, Array[(Long, Double)])] has been found out dynamically at run time based on another RDD of same type. So is there a way to get this value on runtime from an RDD ?

Answer 1

You can use Scala's TypeTag s

import scala.reflect.runtime.universe._
def checkEqualParameters [T1, T2](x : T1, y : T2)(implicit type1 : TypeTag[T1], type2 : TypeTag[T2]) = { 
    type1.tpe.typeArgs == type2.tpe.typeArgs 
}

And then compare

checkEqualParameters (rdd1, rdd2)