Scala / Spark-聚合RDD

Question

Just wondering how I can do the following:

Suppose I have an RDD containing (username, age, movieBought) for many usernames and some lines can have the same username and age but a different movieBought.

How can I remove the duplicated lines and transform it into (username, age, movieBought1, movieBought2...)?

Kind Regards

Answer 1

val grouped = rdd.groupBy(x => (x._1, x._2)).map(x => (x._1._1, x._1._2, x._2.map(_._3)))

val results = grouped.collect.toList

UPDATE (if each tuple also has number of movies item):

val grouped = rdd.groupBy(x => (x._1, x._2)).map(x => (x._1._1, x._1._2, x._2.map(m => (m._3, m._4))))

val results = grouped.collect.toList

Answer 2

I was gonna suggest collect and to list, but ka4eli beat me to it.

I guess you could also use the groupBy / groupByKey and then reduce/reduceByKey operation. The downside of this ofc is that the result (movie1,movie2,movie3..) are concatenated into 1 string (instead of a List structure, which makes accessing it difficult).

val group = rdd.map(x=>((x.name,x.age),x.movie))).groupBy(_._1)
val result =  group.map(x=>(x._1._1,x._1._2,x._2.map(y=>y._2).reduce(_+","+_)