Scala / Spark减少类对象的RDD

Question

I need your help for my last step of a school project.

val conf: SparkConf = new SparkConf() .setMaster("local[*]") .setAppName("AppName") .set("spark.driver.host", "localhost")
    val sc: SparkContext = new SparkContext(conf)
    var list_creature = new ListBuffer[creature]()
    list_creature += new creature("ska")
    list_creature(0).addspell("Heal")
    list_creature(0).addspell("Attaque")
    list_creature += new creature("moise")
    list_creature(1).addspell("Tank")
    list_creature(1).addspell("Defense")
    list_creature(1).addspell("Attaque")
    val rdd = sc.parallelize(list_creature)
    val y = rdd.map(e=>(e.name,e.Spells)).collect()
    val z = y.flatMap(x =>ListBuffer(x._2->x._1))
    val ze = z.flatMap(e =>e._1.flatMap(x => ListBuffer(x->e._2)))

i get this as a result,

(Heal,ska)
(Attaque,ska)
(Tank,moise)
(Defense,moise)
(Attaque,moise)

So, i want to reduce this List[List[String]] to get List[String,List[string]] and the result will be :

(Heal,(ska))
(Attaque,(ska,moise))
(Tank,(moise))
(Defense,(moise))

Thanks you're the best ...

Answer 1

Not sure why you create a RDD then collect before all the major transformations. Since you didn't provide definition of class Creature , I'm creating a placeholder class based on your question content as follows:

class Creature(val name: String) extends Serializable {
  var spells: List[String] = List.empty[String]
  def addspell(spell: String): Unit = {
    spells ::= spell
  }
}

import scala.collection.mutable.ListBuffer

val list_creature = ListBuffer[Creature]()

list_creature += new Creature("ska")
list_creature(0).addspell("Heal")
list_creature(0).addspell("Attaque")

list_creature += new Creature("moise")
list_creature(1).addspell("Tank")
list_creature(1).addspell("Defense")
list_creature(1).addspell("Attaque")

val rdd = sc.parallelize(list_creature)

val reducedRDD = rdd.flatMap( c => c.spells.map(s => (s, List(c.name))) ).
  reduceByKey( _ ++ _ )

reducedRDD.collect
// res1: Array[(String, List[String])] = Array(
//   (Heal,List(ska)), (Defense,List(moise)), (Attaque,List(ska, moise)), (Tank,List(moise)
// ))