Scala，RDD数组[字符串]连接

Question

Is there any way I could concatenate three RDD Array[string]? I am new to Scala and I'm still learning new technique.

I have three RDD Array[string] that looks like this:

    RDD1 = ['string1', 'string2', 'string3']
    RDD2 = ['stringa', 'stringb', 'stringc']
    RDD3 = ['stringA', 'stringB', 'stringC']

But the trick is, I need to add first columns into first row. So, after concatenation, it should look like this:

RDD = ['string1', 'stringa', 'stringA'
       'string2', 'stringb', 'stringB'
       'string3', 'stringc', 'stringC']

if i use .union

that would just give me this:

['string1', 'string2', 'string3'
 'stringa', 'stringb', 'stringc',
 'stringA', 'stringB', 'stringC']

is there any way to accomplish this?

Answer 1

In regular scala, you could do it with transpose , like:

Array(r1, r2, r3).transpose.flatten

I'm not very familiar with Spark, but I don't believe transpose is available. If you know you just need a 3x3, you can get the same result with:

r1 zip r2 zip r3 flatMap {case ((a, b), c) => Array(a,b,c)}

If you need to generalize to any nxn , that's going to require a recursive algorithm.

Answer 2

So you want the first row of three rdds to be together. You can do that easily by first doing' zipwithindex' and join based on the index of three of your rdds. I am assuming you want them to be in same record because rdd don't have a sense of ordering in them.