如何从 Spark Scala 中的 Case 类映射数组

Question

Sample Data :( 251~jhon~WrappedArray([STD,Health,Duval]) )

case class xyz(id : String, code : String, County : String)

case class rewards(memId : String, name: String, part: Array[xyz])

val df = spark.read.textFile("file:///data/").rdd.map(r => r.split('~'))

val df2 = df.map(x => { rewards(x(0),x(1), Array[rewards.apply()] ) } )

tried many way to map an array from case class. tried apply function

Answer 1

I am not sure that's what you are looking for but you can try using pattern matching to transform arrays into case classes.

val data: RDD[rewards] = sc
    .parallelize(Seq("251~jhon~WrappedArray([STD,Health,Duval])"))
    .map(_.split("~"))
    .map{ case Array(id, code, part) => (id, code, part
               .replaceFirst("\\s*WrappedArray\\(\\s*\\[\\s*", "")
               .replaceFirst("\\s*\\]\\s*\\)\\s*", "") 
    )}
    .map{ case (id, name, part) => rewards(id, name, part.split("\\s*,\\s*") match {
         case Array(id, code, county) => Array(xyz(id, code, county)) 
    })}