error: value join is not a member of org.apache.spark.rdd.RDD[(Long, U)]?

Question

I'm trying to write the following method:

scala>   def isEqual[U, V](expected: RDD[U], result: RDD[V]) = {
 |         val expectedIndexValue: RDD[(Long, U)] = expected.zipWithIndex().map{ case (row, idx) => (idx, row) }
 |         val resultIndexValue: RDD[(Long, V)] = result.zipWithIndex().map{ case (row, idx) => (idx, row) }
 |         val combined = expectedIndexValue.join(resultIndexValue)
 |       }

But I got the following error:

<console>:52: error: value join is not a member of org.apache.spark.rdd.RDD[(Long, U)]
         val combined = expectedIndexValue.join(resultIndexValue)

Answer 1

The join function is defined on a special type called PairRDDFunctions and there is an implicit conversion between RDDs of tuples of this special type. If you are working in an old version of Spark you would need to import the implicit conversions eg import org.apache.spark.SparkContext._ . There is also an explicit (albiet deprecated) function on the SparkContext called rddToPairRDDFunctions that you can manually use.

However both the explicit function and the implicit conversions require that the class tags for the types be present, so you could use a fake class tag, but since you're working in Scala you might as well just add the class tags as implicit params to your function.