将双RDD保存到文件-Scala

Question

I am trying to save in a file a double RDD, what I mean by a double RDD is that I have this variable:

res: org.apache.spark.rdd.RDD[org.apache.spark.rdd.RDD[((String,String), Int)]] = MapPartitionsRDD[19] 

I tried to store it with

res.saveAsTextFile(path)

But it doesn't work, an exception is launched because Spark does not support nested RDD here is a sample of the code:

val res = Listword.map { x =>
Listword.map { y =>
  ((x._1, y._1), x._2 + y._2)
}
}
res.saveAsTextFile("C:/Users/Administrator/Documents/spark/spark-1.6.0-bin-hadoop2.6")

Answer 1

Spark does not allow nested RDDs. In your specific case, you can use cartesian :

ListWord.cartesian(ListWord).map { case (x, y) =>
  ((x._1, y._1), x._2 + y._2)
}