将元素合并到rdd中的数组

Question

how can I convert an RDD[(Int,Int)] to an RDD[Array[(Int,Int)]] where I combine elements with their key.

Lets say

(0,0),(1,0),(1,1),(0,1)

and I want it to be an Array arr1 = ((0,0),(1,0)) and an arr2 ((1,1),(0,1)) So the resulted rdd will have arr1,arr2 as arrays.

Answer 1

What you're basically trying to do is group an RDD[TupleN] by the i th element. You can use

rdd.groupBy(_._1)

to create a

Map[T, RDD[TupleN]]

where the key will be the i th element (ie, 0 or 1 in your example).

Then you can map the values of this map to an array with mapValues(_.toArray)