基于顶点ID创建边缘会触发scala

Question

I want to basically join two RDDs vertices and edges. Vertices and edges are created using the following code:

val file = sc.textFile("file.gz") //This tab separated file has more than two columns among which only first two columns with source and destination URL are relevant 

val edges= file.flatMap(f => {
  val urls = f.split("\t")
  if (!(urls.length < 2)) 
{ Some(urls(0) +"\t"+ urls(1)) }
else None }).distinct

val vertices = edges.flatMap(f => f.split("\t")).distinct 
val vertices_zip = vertices.zipWithUniqueId

Now I have a list of vertices (URLs) with IDs generated using using the above method like:

google.de/2011/10/Extract-host,11
facebook.de/2014/11/photos,28         
community.cloudera.com/t5/,42         
facebook.de/2020/11/photos,91 

I would like to create edges based on these IDs. Edges RDD file is tab separated like below:

google.de/2011/10/Extract-host   facebook.de/2014/11/photos   
facebook.de/2014/11/photos       community.cloudera.com/t5/,42
community.cloudera.com/t5/       google.de/2011/10/Extract-host

Required result:

11     28
28     42
42     11

I tried the following code

val edges_id = edges.flatMap( line => line.split( "\t" ) ).map( line => ( line,0) ) .join(vert_zip).map(x=>x._2._2)

But not getting the desired result. I am getting

11
28
28
42
42
11

I am not sure how to join the edges with the vertices RDD to get this result. Any help would be much appreciated.

Answer 1

When you zipWithUniqueId , then collect the rdds as map and then use that map to get the indexes in the edges rdd as following

val vertices_zip = vertices.zipWithUniqueId.collectAsMap

val edges_id = edges.map(f => {
  val urls = f.split("\t")
  vertices_zip(urls(0))+"\t"+vertices_zip(urls(1))
})

Thats all. I hope the answer is helpful

Updated

you commented

I am getting an exception : java.lang.OutOfMemoryError: Java heap space

for that you can use broadcast which would call required rdds to the executors memory instead of all the map

val vertices_zip = sc.broadcast(vertices.zipWithUniqueId.collectAsMap)

val edges_id = edges.map(f => {
  val urls = f.split("\t")
  vertices_zip.value(urls(0))+"\t"+vertices_zip.value(urls(1))
})

joins

You've commented again

Is it possible to change the code I tried above to get the result (the one with the join)?

join way would require two joins meaning that two shuffles would be needed to get the desired result

val vertices_zip = vertices.zipWithUniqueId

val edges_id = edges.map(line => {
  val splitted = line.split("\t")
  (splitted(0), splitted(1))
})
  .join(vertices_zip)
  .map(_._2)
  .join(vertices_zip)
  .map(x => x._2._1+"\t"+x._2._2)