如何从包含顶点和边的文本文件创建图形？

Question

I have created an RDD of two input files ie Edges and Node files. While I use the Graph.fromEdge() method to create a graph, I get errors. Could someone please help me? The inputEdgesTextFile and inputNodesTextFile are taking the input text dataset. On the very last line of the code, I am getting error. I am posting an error that I am getting in my code.

public static void main(String[] args) {

    SparkConf conf = new SparkConf().setMaster("local").setAppName("GraphFileReadClass");
    JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
    ClassTag<String> stringTag = scala.reflect.ClassTag$.MODULE$.apply(String.class);
    ClassTag<String> intTag = scala.reflect.ClassTag$.MODULE$.apply(Integer.class);

    $eq$colon$eq<String, String> tpEquals = scala.Predef.$eq$colon$eq$.MODULE$.tpEquals();
    // Load an external Text File in Apache spark
    //The text files number of lines and each line consists these structure
    //SFEdge contains: | Edge_id integer | Source_Id integer | Destination_id integer | EdgeLength double |
    //SFNodes contains: | Node_id integer | Longitude double | Latitude double |
    
    
    JavaRDD<String> inputEdgesTextFile = javaSparkContext.textFile("./SFEdges.txt");
    JavaRDD<String> inputNodesTextFile = javaSparkContext.textFile("./SFNodes.txt");
    ArrayList<Tuple2<Integer, Integer>> nodes = new ArrayList<>();
    ArrayList<Edge<Double>> edges = new ArrayList<>();

    JavaRDD<NodesClass> nodesPart = inputNodesTextFile.mapPartitions(p -> {
        ArrayList<NodesClass> nodeList = new ArrayList<NodesClass>();
        int counter = 0;
        while (p.hasNext()) {
            String[] parts = p.next().split(" ");
            NodesClass node = new NodesClass();
            node.setNode_Id(Integer.parseInt(parts[0]));
            node.setLongitude(Double.parseDouble(parts[1]));
            node.setLatitude(Double.parseDouble(parts[2]));
            nodes.add(new Tuple2<Integer, Integer>(counter, Integer.parseInt(parts[0])));
            nodeList.add(node);
            counter++;

        }
        return nodeList.iterator();
    });
    JavaRDD<Tuple2<Integer, Integer>> nodesRDD = javaSparkContext.parallelize(nodes);
    nodesRDD.foreach(data -> System.out.print("Node details: " + data._1() + " " + data._2()));

    JavaRDD<EdgeNetwork> edgesPart = inputEdgesTextFile.mapPartitions(p -> {
        ArrayList<EdgeNetwork> edgeList = new ArrayList<EdgeNetwork>();
        while (p.hasNext()) {

            String[] parts = p.next().split(" ");
            EdgeNetwork edgeNet = new EdgeNetwork();
            edgeNet.setEdge_id(Integer.parseInt(parts[0]));
            edgeNet.setSource_id(Integer.parseInt(parts[1]));
            edgeNet.setDestination_id(Integer.parseInt(parts[2]));
            edgeNet.setEdge_length(Double.parseDouble(parts[3]));
            edges.add(new Edge<Double>(Long.parseLong(parts[1]), Long.parseLong(parts[2]),
                    Double.parseDouble(parts[3])));
            edgeList.add(edgeNet);

        }
        return edgeList.iterator();
    });
    JavaRDD<Edge<Double>> edgesRDD = javaSparkContext.parallelize(edges);

    Graph<String, Double> graph = Graph.fromEdges(edgesRDD.rdd(), " ", StorageLevel.MEMORY_ONLY(),
            StorageLevel.MEMORY_ONLY(), stringTag, stringTag);
    //The warning shows above this line for Graph<String,Double>
    //Maybe the RDD that I have created has some errors. Please suggest me

Answer 1

Graph.fromEdges looks in Scala like

def fromEdges[VD: ClassTag, ED: ClassTag](
    edges: RDD[Edge[ED]],
    defaultValue: VD,
    edgeStorageLevel: StorageLevel = StorageLevel.MEMORY_ONLY,
    vertexStorageLevel: StorageLevel = StorageLevel.MEMORY_ONLY): Graph[VD, ED] = {
...

The two class tags are translated in the Java-API as additional parameters at the end of the method. VD is a string type here and ED is a double type, so the Java call should reflect these types, the second class tag should be Double:

ClassTag<Double> doubleTag = scala.reflect.ClassTag$.MODULE$.apply(Double.class);

Graph.fromEdges(edgesRDD.rdd(), " ", StorageLevel.MEMORY_ONLY(),
        StorageLevel.MEMORY_ONLY(), stringTag, doubleTag);