I trying to find algorithm of searching disjoint sets (connected components/union-find) on large amount of data with apache spark. Problem is amount of data. Even Raw representation of graph vertex doesn't fit in to ram on single machine. Edges also doesn't fit in to the ram.
Source data is text file of graph edges on hdfs: "id1 \\t id2".
id present as string value, not int.
Naive solution that I found is:
[id1:id2] [id3:id4] [id1:id3]
[id1:[id2;id3]][id3:[id4]]
(flatMap) [id1:id1][id2:id1][id3:id1][id3:id3][id4:id3]
[id2:id1] -> [id1:id2]
leftOuterJoin
of rdds from stage 3 and 4 But this results in the transfer of large amounts of data between nodes (shuffling)
Any advices?
If you are working with graphs I would suggest that you take a look at either one of these libraries
They both provide the connected components algorithm out of the box.
GraphX :
val graph: Graph = ...
val cc = graph.connectedComponents().vertices
GraphFrames :
val graph: GraphFrame = ...
val cc = graph.connectedComponents.run()
cc.select("id", "component").orderBy("component").show()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.