How can I make my Dijkstra algorithm more efficient

Question

I have a directed network model made up of a set of nodes connected by links that grows with each model iteration. In order to find the "average shortest path" in the final model iteration, I have implemented Dijkstra's algorithm that calculates the shortest path from all nodes to all nodes. To be more specific, the algorithm calculates the shortest path from each of the networks 3,000 nodes to all other 3,000 nodes (if a path exists), roughly 9,000,000 pathlengths and then finds the average path length. When I try this, I run out of memory. I am able to get an average path length up until about 500 nodes, where roughly 250,000 path lengths are calculated in under 12h. My question is, is there any way to improve the code in a way that might make it more efficient? Or is it not feasible to calculate that many paths?

Code below... the algorithm itself is adapted from Vogella http://www.vogella.com/tutorials/JavaAlgorithmsDijkstra/article.html

Nodes in the nework represent trees and edges or links represent nets.

Dijstra Algorithm

package network;

imports...

public class DijkstraAlgorithm {
    private Context<Object> context;
    private Geography<Object> geography;
    private int id; 
    List<Tree> vertices = Tree.getVertices();
    List<Nets> edges = Nets.getEdges();
    private Set<Tree> settledNodes;
    private Set<Tree> unSettledNodes;
    private Map<Tree, Tree> predecessors;
    private Map<Tree, Integer> distance;

public DijkstraAlgorithm(Graph graph) {
    this.context = context;
    this.geography = geography;
    this.id = id;
    this.vertices = vertices;
    this.edges = edges;
}


setters and getters....

public void execute(Tree source){
    settledNodes = new HashSet<Tree>();
    unSettledNodes = new HashSet<Tree>();
    distance = new HashMap<Tree, Integer>();
    predecessors = new HashMap<Tree, Tree>();
    distance.put(source, 0);
    unSettledNodes.add(source);
    while (unSettledNodes.size()>0){
        Tree node = getMinimum(unSettledNodes);
        settledNodes.add(node);
        unSettledNodes.remove(node);
        findMinimalDistances(node);
    }
}

private void findMinimalDistances(Tree node){
    List<Tree>adjacentNodes = getNeighbors(node);
    for (Tree target: adjacentNodes){
        if (getShortestDistance(target)>getShortestDistance(node)+getDistance(node,target)){
            distance.put(target, getShortestDistance(node) + getDistance(node, target));
            predecessors.put(target, node);
            unSettledNodes.add(target);
        }

    }
}

private int getDistance(Tree node, Tree target){
    for (Nets edge: edges){
        if (edge.getStartTrees().equals(node) && edge.getEndTrees().equals(target)){
            return edge.getId();
        }
    }
    throw new RuntimeException("Should not happen");
}

private List<Tree> getNeighbors(Tree node){
    List<Tree> neighbors = new ArrayList<Tree>();
    for (Nets edge: edges) {
        if(edge.getStartTrees().equals(node) && !isSettled(edge.getEndTrees())){
            neighbors.add(edge.getEndTrees());
        }
    }
    return neighbors;
}

private Tree getMinimum(Set<Tree>vertexes){
    Tree minimum = null;
    for (Tree vertex: vertexes) {
        if (minimum == null){
            minimum = vertex;
        } else {
            if (getShortestDistance(vertex)< getShortestDistance(minimum)){
                minimum = vertex;
            }
        }
    }

    return minimum;

}

private boolean isSettled(Tree vertex){
    return settledNodes.contains(vertex);
}

private int getShortestDistance(Tree destination) {
    Integer d = distance.get(destination);
    if (d == null) {
        return Integer.MAX_VALUE;
    } else {
        return d;
    }
}

public LinkedList<Tree> getPath(Tree target){
    LinkedList<Tree>path = new LinkedList<Tree>();
    Tree step = target;
    if(predecessors.get(step)== null){
        return null;
    }
    path.add(step);
    while (predecessors.get(step)!=null){
        step = predecessors.get(step);
        path.add(step);

    }
    Collections.reverse(path);
    return path;
}



}

Graph

package network;

imports...

public class Graph {
     private Context<Object> context;
     private Geography<Object> geography;
     private int id; 
     List<Tree> vertices = new ArrayList<>();
     List<Nets> edges = new ArrayList<>();
     List <Integer> intermediateNodes = new ArrayList<>();

public Graph(Context context, Geography geography, int id, List vertices, List edges) {
    this.context = context;
    this.geography = geography;
    this.id = id;
    this.vertices = vertices;
    this.edges = edges;
}

setters... getters...

//updates graph
@ScheduledMethod(start =1, interval =1, priority =1)
public void countNodesAndVertices() {
    this.setVertices(Tree.getVertices());
    this.setEdges(Nets.getEdges());

//run Dijkstra at the 400th iteration of network development
@ScheduledMethod(start =400, priority =1)
public void Dijkstra(){

    Graph graph2 = new Graph (context, geography, id, vertices, edges);
    graph2.setEdges(this.getEdges());
    graph2.setVertices(this.getVertices());
    for(Tree t: graph2.getVertices()){
    }
    DijkstraAlgorithm dijkstra = new DijkstraAlgorithm(graph2);

    //create list of pathlengths (links, not nodes)
    List <Double> pathlengths = new ArrayList<>();
    //go through all nodes as starting nodes
    for (int i = 0; i<vertices.size();i++){

        //find the shortest path to all nodes as end nodes
        for (int j = 0; j<vertices.size();j++){
            if(i != j){

                Tree startTree = vertices.get(i);
                Tree endTree = vertices.get(j);
                dijkstra.execute(vertices.get(i));
                //create a list that contains the path of nodes
                LinkedList<Tree> path = dijkstra.getPath(vertices.get(j));
                     //if the path is not null and greater than 0
                    if (path != null && path.size()>0){
                    //calculate the pathlength (-1, which is the size of the path length of links) 
                    double listsize = path.size()-1;
                    //add it to the list
                    pathlengths.add(listsize);
                }

            }
        }



    }   
    calculateAvgShortestPath(pathlengths);

}
//calculate the average
public void calculateAvgShortestPath(List<Double>pathlengths){
    Double sum = 0.0;
    for (Double cc: pathlengths){
        sum+= cc;
    }
    Double avgPathLength = sum/pathlengths.size();
    System.out.println("The average path length is: " + avgPathLength);

}

}

Answer 1

One quick improvement is to move the line:

dijkstra.execute(vertices.get(i));

up 6 lines (so it is in the i loop, but not the j loop).

This should improve runtime by the number of nodes (ie 3000 times faster).

It should still give identical results because Dijkstra's algorithm calculates the shortest path from the start node to ALL destination nodes, so there is no need to rerun it for each pair of start/end.

Answer 2

There are several optimizations that you could make. Like for instance using a Fibonacci heap (or even a standard java priority queue) would definitely speed things up. However, the memory issue will likely persist for a dataset that large regardless. The only real way to deal with a dataset that big is to use a distributed implementation. I believe that there is a shortest path implementation in the Spark Graphx library that you could use.