Spark GraphX的DFS性能与简单的Java DFS实现

Question

Considering a graph with 14,000 vertices and 14,000 edges, I wonder why GraphX takes much more time than the java implementation of a graph to get all the paths from a vertex to the leaf?

The java implementation: A few seconds

The Graphx implementation: Several minutes

Is spark GraphX really suitable for this kind of treatment?

My system: i5-7500 @3.40GHz, 8GB RAM

The pregel's algorythm:

val sourceId: VertexId = 42 // The ultimate source
  // Initialize the graph such that all vertices except the root have canReach = false.
  val initialGraph: Graph[Boolean, Double]  = graph.mapVertices((id, _) => id == sourceId)
  val sssp = initialGraph.pregel(false)(
    (id, canReach, newCanReach) => canReach || newCanReach, // Vertex Program
    triplet => {  // Send Message
      if (triplet.srcAttr && !triplet.dstAttr) {
        Iterator((triplet.dstId, true))
      } else {
        Iterator.empty
      }
    },
    (a, b) => a || b // Merge Message

Answer 1

It happened to me when implementing some algorithms on Graphx, I believe that GraphX is well adapted for a distributed environment when you have big graphs split accross multiple machines. But now while you say that you use one node, have you checked the number of workers used? number of executors? Amount of memory used by each excutor? These configuration parameters definitely plays an important role in increasing or decreasing the performance of your GraphX application.