Spark join——（边和顶点）

Question

I have vertexRDD which has 2 columns

(vertexId, uniqueVertexId)
(V1, 1L)
(V2, 2L)
(V3, 3L)
(V4, 4L)

And I also have edgeRDD

(srcId, destId)
(V1, V2)
(V2, V3)
(V1, V4)

How can I join them in spark so the edges RDD will be like below

(srcId, destId, uniqueSrcId, uniqueDestId)
(V1, V2, 1L, 2L)
(V2, V3, 2L, 3L)
(V1, V4, 1L, 4L)

I tried different joins but I couldn't really achieve the expected output. Appreciate any help.

Answer 1

I will use Java but I guess it is straightforward to convert it to Scala.
Assuming
edgeRDD has type JavaPairRDD<String,String> and
vertexRDD has type JavaPairRDD<String,Long> :

1. edgeRDD.join(vertexRDD) will yield JavaPairRDD<String,Tuple2<String,Long>> with the following content (let's call it join1 ):

    (V1, Tuple2(V2,1L)) (V2, Tuple2(V3,2L)) (V1, Tuple2(V4,1L))

2. Then you convert join1 into another JavaPairRDD<String,Tuple2<String,Long>> by restructuring the keys and values using map (let's call it join2 ):

    (V2, Tuple2(V1,1L)) (V3, Tuple2(V2,2L)) (V4, Tuple2(V1,1L))

3. Finally perform vertexRDD.join(join2) to get JavaPairRDD<String,Tuple2<Long,Tuple2<String,Long>>> with contents:

    (V2, Tuple2(2L, Tuple2(V1,1L))) (V3, Tuple2(3L, Tuple2(V2,2L))) (V4, Tuple2(4L, Tuple2(V1,1L)))

which you may pass through the map and create JavaRDD<String> (or a new JavaPairRDD ) by combining keys and values appropriately within the map. I will leave mapping phases up to you.