Spark如何将RDD [Seq [（String，String）]]转换为RDD [（String，String）]

Question

I have a Spark RDD[Seq[(String,String)]] which contains several group of two words. Now I have to save them to a file in HDFS like this (no matter in which Seq they are):

dog cat
cat mouse
mouse milk

Could someone help me with this? Thanks a lot <3

EDIT : Thanks for your help. Here is the solution

Code

val seqTermTermRDD: RDD[Seq[(String, String)]] = ...
val termTermRDD: RDD[(String, String)] = seqTermTermRDD.flatMap(identity)
val combinedTermsRDD: RDD[String] = termTermRDD.map{ case(term1, term2) => term1 + " " + term2 }
combinedTermsRDD.saveAsTextFile(outputFile)

Answer 1

RDDs have a neat function called "flatMap" that will do exactly what you want. Think of it as a Map followed by a Flatten (except implemented a little more intelligently)--if the function produces multiple entities, each will be added to the group separately. (You can also use this for many other objects in Scala.)

val seqRDD = sc.parallelize(Seq(Seq(("dog","cat"),("cat","mouse"),("mouse","milk"))),1)
val tupleRDD = seqRDD.flatMap(identity)
tupleRDD.collect  //Array((dog,cat), (cat,mouse), (mouse,milk))

Note that I also use the scala feature identity , because flatMap is looking for a function that turns an object of the RDD's type to a TraversableOnce , which a Seq counts as.

Answer 2

You can also use mkString( sep ) function ( where sep is for separator) on Scala collections. Here are some examples: (note that in your code you would replace the last .collect().mkString("\\n") with saveAsTextFile(filepath) ) to save to Hadoop.

scala> val rdd = sc.parallelize(Seq(  Seq(("a", "b"), ("c", "d")),  Seq( ("1", "2"), ("3", "4") )      ))
rdd: org.apache.spark.rdd.RDD[Seq[(String, String)]] = ParallelCollectionRDD[6102] at parallelize at <console>:71

scala> rdd.map( _.mkString("\n")) .collect().mkString("\n")
res307: String = 
(a,b)
(c,d)
(1,2)
(3,4)

scala> rdd.map( _.mkString("|")) .collect().mkString("\n")
res308: String = 
(a,b)|(c,d)
(1,2)|(3,4)

scala> rdd.map( _.mkString("\n")).map(_.replace("(", "").replace(")", "").replace(",", " ")) .collect().mkString("\n")
res309: String = 
a b
c d
1 2
3 4