我想将org.apache.spark.rdd.RDD [（Any，scala.collection.immutable.Iterable [String]）]展平为Scala中的String

Question

Example

(168,20874,List(, 33895, 2711))

to 168,20874| , 33895, 2711

Basically convert RDD[(Any, scala.collection.immutable.Iterable[String])] to String.

Thanks

Answer 1

Method "map" with predefined formatter can be used:

val rdd = sparkContext.parallelize(List((168, 20874, List(33895, 2711))))
val result = rdd.map { case (a, b, c) => s"$a,$b| ,${c.mkString(",")}" }
result.foreach(println)

Output:

168,20874| ,33895,2711