GraphFrame 最短路径因未找到 Vertex Id 错误而失败

Question

My Spark/Databricks is using GraphFrames to find shortest paths between vertices inside a connected component. Algorithm is failing after several minutes with org.graphframes.NoSuchVertexException: GraphFrame algorithm given vertex ID which does not exist in Graph. Vertex ID 1 not contained in GraphFrame org.graphframes.NoSuchVertexException: GraphFrame algorithm given vertex ID which does not exist in Graph. Vertex ID 1 not contained in GraphFrame .

The error message is completely irrelevant - neither graph's vertices contain id == 1 nor edges src dst does. The algorithm should not be looking for such id at all. I'm wondering if there is any size driven limitation causing shortestPaths failure or if I'm missing any other part of definition.

Code is very simple:

val sp14754224 = g54.shortestPaths.landmarks("14754224").run

Graph structure quite basic too:

e54:org.apache.spark.sql.DataFrame
    src:integer
    dst:integer
    edgeRevenue:double
    edgeAgreements:double

v54:org.apache.spark.sql.DataFrame
    id:integer
    Name:string
    vertexRevenue:double
    vertexDealss:long

Graph itself is relatively large (31,342 vertices and 1,027,724 edges), but it's just subset of larger graph previously processed by connectedComponets . There also seems to be no issues with memory consumption (observed peak was ~20GB while each worker has 64GB).

Any recommendation?

Answer 1

I believe landmarks is supposed to be a sequence, try this:

val sp14754224 = g54.shortestPaths.landmarks(Seq("14754224")).run

I wonder if there is some conversion going on so your string is becoming a Seq[Char] perhaps, hence the vertex 1 error.

Answer 2

Never found solution in Spark/Scala, but there is an easy work-around using Spark R:

• Switch to R (either created R notebook or use %r ; either way, input data has to be read from the storage)
• Install iGraph and SparkR libraries to the cluster
• Collect vertices and edges to get R data.frame
• Apply iGraph methods (eg shortest path)

Graph data must fit in driver's memory which was the case for the task from my question.

Code Example:

%r
if (require(SparkR) == FALSE) install.packages("SparkR")
if (require(igraph) == FALSE) install.packages("igraph")

processRoot <-  "abfss://yourAccount@fdestorageuat.dfs.core.windows.net/YourDataPath/"

#Data Intake
inEdgesPath     <- paste(processRoot, "STG_Edges/", sep = "")
inVerticesPath  <- paste(processRoot, "STG_Vertices/", sep = "")

inEdges     <- collect(read.parquet(inEdgesPath))
inVertices  <- collect(read.parquet(inVerticesPath))

#Graph Definition (inVertices is optional - adding to capture names size of vertices)
g <- graph_from_data_frame(validEdges, directed = FALSE, inVertices)

#Example of iGraph (defining connected components and communities)
clu <- components(g)
fg <- fastgreedy.community(g)

#Outputs from both commands have same order and lenght as vertices thus they could be added to vertices data
inVertices$componentId <- clu[["membership"]]
inVertices$communityId <- as.numeric(membership(fg))

iGraph has much broader functionality than GraphX/GraphFrame. It also performs much faster as long as graph data fits in memory. If graph should be too large, consider using GraphFrame's connected components first and then process each component separately calling iGraph's functionality thru gapply by each component id.