无法在GraphX（Scala Spark）中创建图形

Question

I have huge problems creating a simple graph in Spark GraphX. I really don't understand anything so I try everything that I find but nothing works. For example I try to reproduce the steps from here .

The following two were OK:

val flightsFromTo = df_1.select($"Origin",$"Dest")

val airportCodes = df_1.select($"Origin", $"Dest").flatMap(x => Iterable(x(0).toString, x(1).toString))

But after this I obtain an error:

val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash.stringHash(x), x))

Error: missing Parameter type

Could You please tell me what is wrong?

And by the way, why MurmurHash? What is a purpose of it?

Answer 1

My guess is that you are working from a 3 year old tutorial with a recent Spark version. The sqlContext read returns a Dataset instead of RDD. If you want it like the tutorial use .rdd. instead

val airportVertices: RDD[(VertexId, String)] = airportCodes.rdd.distinct().map(x => (MurmurHash3.stringHash(x), x))

or change type of variable

val airportVertices: Dataset[(Int, String)] = airportCodes.distinct().map(x => (MurmurHash3.stringHash(x), x))

You could also checkout https://graphframes.github.io/ if you are interested in Graphs and Spark

---

Updated

To create a Graph you need vertices and edges To make computation easier all vertices have to be identified by a VertexId (in essence a Long)

The MurmerHash is used to create very good unique hashes. More info here: MurmurHash - what is it?

Hashing is a best practise to distribute the data without skewing, but there is no technical reason why you couldn't use an incremental counter for each vertex

I've looked at the tutorial, but the only thing you have to change to make it work, is to add .rdd :

val flightsFromTo = df_1.select($"Origin",$"Dest").rdd
val airportCodes = df_1.select($"Origin", $"Dest").flatMap(x => Iterable(x(0).toString, x(1).toString)).rdd