将RDD Array [Any] = Array（List（[String]，ListBuffer（[string]））转换为RDD（String，Seq [String]）

Question

I have a RDD with Any type, example:

Array(List(Mathematical Sciences, ListBuffer(applications, asymptotic, largest, enable, stochastic)))

I want to convert it to RDD of type RDD[(String, Seq[String])]

I tried:

val rdd = sc.makeRDD(strList)
case class X(titleId: String, terms: List[String])

val df = rdd.map { case Array(s0, s1) => X(s0, s1) }.toDF()

I passed a long time to try without success

Answer 1

You can use:

val result: RDD[(String, Seq[String])] = 
  rdd.map { case List(s0: String, s1: ListBuffer[String]) =>  (s0, s1) }

But note that any record in the input RDD[Any] that doesn't match these types (that can't be checked in compile time) would throw a scala.MatchError .

Answer 2

As mentioned in the question, if you have

val strList = Array(List("Mathematical Sciences", ListBuffer("applications", "asymptotic", "largest", "enable", "stochastic")))
val rdd = sc.makeRDD(strList)

which is of following dataTypes

rdd: org.apache.spark.rdd.RDD[List[java.io.Serializable]]

You can convert it to your required dataTypes

res0: org.apache.spark.rdd.RDD[(String, Seq[String])]

by simply using map and converting the dataTypes as

rdd.map(x => (x(0).toString, x(1).asInstanceOf[ListBuffer[String]].toSeq))

I hope the answer is helpful

Answer 3

Finally , it s worked i have a warning but worked

val rdd = sc.makeRDD(strList)

val result = rdd.map { case List(s0: String, s1: Seq[String]) => (s0, s1) }

:32: warning: non-variable type argument String in type pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated by erasure val result = rdd.map { case List(s0: String, s1: Seq[String]) => (s0, s1) } ^ result: org.apache.spark.rdd.RDD[(String, Seq[String])] = MapPartitionsRDD[1051] at map at :32

thank you