spark：如何用另一个RDD的每个分区压缩一个RDD

Question

Let's say I have one RDD[U] that will always consist of only 1 partition. My task is fill this RDD up with the contents of another RDD[T] that resides over n number of partitions. The final output should be n number of partitions of RDD[U] .

What I tried to do originally is:

val newRDD = firstRDD.zip(secondRDD).map{ case(a, b)  => a.insert(b)}

But I got an error: Can't zip RDDs with unequal numbers of partitions

I can see in the RDD api documentation that there is a method called zipPartitions() . Is it possible, and if so how, to use this method to zip each partition from RDD[T] with a single and only partition of RDD[U] and perform a map on it as I tried above?

Answer 1

Something like this should work:

val zippedFirstRDD = firstRDD.zipWithIndex.map(_.swap)
val zippedSecondRDD = secondRDD.zipWithIndex.map(_.swap)

zippedFirstRDD.join(zippedSecondRDD)
  .map{case (key, (valueU, valueT)) => {
    valueU.insert(valueT)
  }}