如何在 spark scala 中加入 2 rdd

Question

I have 2 RDD's as below

val rdd1 = spark.sparkContext.parallelize(Seq((123, List(("000000011119",20),("000000011120",30),("000000011121",50))),(234, List(("000000011119",20),("000000011120",30),("000000011121",50)))))
val rdd2 = spark.sparkContext.parallelize(Seq((123, List("000000011119","000000011120")),(234, List("000000011121","000000011120"))))

I want to perform addition of values in rdd1 on the basis of key pairs in rdd2.

Output required:

RDD[(123,50),(234,80)]

Any help will be appreciated.

Answer 1

Really this is a join on the first element of the row, and the first element of each of the contents.

So I'd explode it into multiple rows and join that way

val flat1 = rdd1.flatMap(r => r._2.map(e => ((r._1, e._1), e._2))) // looks like ((234,000000011119),20)
val flat2 = rdd2.flatMap(r => r._2.map(e => ((r._1, e), true))) // looks like ((234,000000011121),true)

val res =  flat1.join(flat2)
  .map(r => (r._1._1, r._2._1))  // looks like (123, 30)
  .reduceByKey(_ + _)  // total each key group

Result with a .foreach(println)

scala> :pas
// Entering paste mode (ctrl-D to finish)

flat1.join(flat2)
  .map(r => (r._1._1, r._2._1))  // looks like (123, 30)
  .reduceByKey(_ + _)  // total each key group
  .foreach(println)

// Exiting paste mode, now interpreting.

(123,50)
(234,80)

As usual, this stuff is much simpler using Dataset, so that would be my recommendation for the future though.