value mkString is not a value of org.apache.spark.rdd.RDD[Int]

Question

I changed this line:

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}.take(20).mkString("::")

to:

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}.mkString("::") 

But Eclipse is giving me an error hint: value mkString is not a value of org.apache.spark.rdd.RDD[Int]

What does this error mean?

Answer 1

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}

This returns an org.apache.spark.rdd.RDD[Int] which is not GenTraversableOnce . Although it has a lot of methods defined which makes it like a Scala collection of Int , it is not ( abstract class RDD[T] extends Serializable with Logging ). It's a bit like a promise of a collection Int . You have to poll the collection out before you mkString with the results.

Call .collect() on the RDD[Int] before you perform mkString .

val ratedNum = rows.sortBy(- _._2).map{case (user , ratednum) => ratednum}.collect.mkString("::")

Or you can add an implicit conversion:

implicit def toArray[T](rdd: RDD[T]) = rdd.collect()