在 Scala 中将 RDD[(Int,Int)] 转换为 PairRDD

Question

这个例子有什么问题？

val f = sc.parallelize(Array((1,1),(1,2)))
val p = new org.apache.spark.rdd.PairRDDFunctions[Int,Int](f)

Name: Compile Error
Message:  error: type mismatch;
 found   : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[(Int, Int)]
 required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[(Int, Int)]
       val p = new org.apache.spark.rdd.PairRDDFunctions[Int,Int](f)
                                                                  ^

Answer 1

您的代码在 Spark 2.2.0 上似乎运行良好。

这是 Spark 2.2.0 版中控制台命令的抄本：

scala> val f = sc.parallelize(Array((1,1),(1,2)))
f: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> val p = new org.apache.spark.rdd.PairRDDFunctions[Int,Int](f)
p: org.apache.spark.rdd.PairRDDFunctions[Int,Int] = org.apache.spark.rdd.PairRDDFunctions@6e1d939e

scala> p
res0: org.apache.spark.rdd.PairRDDFunctions[Int,Int] = org.apache.spark.rdd.PairRDDFunctions@6e1d939e

scala> f
res1: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[0] at parallelize at <console>:24

使用 Scala 版本 2.11.8（OpenJDK 64 位服务器 VM，Java 1.8.0_131）

对我来说，这似乎是旧版本中的错误。