有向无环图中所有节点的可达性计数

Question

So there was this challenge on a programming contest on Hackerrank called "Acyclic Graph", which basically boils down to counting the number of nodes reachable from every node in a "Directed Acyclic Graph". For example, say you have a graph like so:

[ 1 ] ---->[ 2 ]--->[ 4 ]--->[ 5 ]
[ 3 ] ------/

Reachability count (including origin node):

Node 1: 4
Node 2: 3
Node 3: 4
Node 4: 2
Node 5: 1

My approach was a "Depth First" traversal with memoization. Looked around quite a bit, but it seems as though the run time can't be improved much further because of the over counting that occurs in cases like so:

[ 1 ] ---->[ 2 ]--->[ 4 ]--->[ 5 ]
[ 3 ] ------/--------/

The third node would count the fourth node, even though the second node already counted the fourth node. To make things a bit worse, I only solve these challenges in JavaScript. Its my primary language and I get a thrill from pushing its boundaries. No one on the leader board has solved it in JavaScript yet, but I presume it's possible. After the contest, I managed to pass 13 out of 24 test cases with the following code:

function Solution( graph, nodes ) {

    var memory = new Array( nodes + 1 )
      , result = 0;

    graph.forEach( ( a, v ) => DepthFirstSearch( graph, v, memory ) );

    // challenge asks for an output variation, but the accurate 
    // reachability count of every node will be contained in "d.length".
    memory.forEach( ( d, i ) => { if ( i && ( 2 * d.length ) >= nodes ) result++; } );

    return result;
}

function DepthFirstSearch( graph, v, memory ) {

    if ( memory[ v ] ) return memory[ v ];

    var descendants = new Uint16Array( [ v ] );

    graph[ v ].forEach( u => {

        descendants = MergeTypedArrays( 
            DepthFirstSearch( graph, u, memory ),  
            descendants
        );
    } );
                                           // make elements unique
                                           // to avoid over counting
    return memory[ v ] = Uint16Array.from( new Set( descendants ) ); 
}

function MergeTypedArrays(a, b) {

    var c = new a.constructor( a.length + b.length );

    c.set( a );
    c.set( b, a.length );

    return c;
}

// adjacency list
var graph = [ 
    [],    // 0
    [ 2 ], // 1
    [ 4 ], // 2
    [ 2 ], // 3
    [ 5 ], // 4
    []     // 5
];

var nodes = 5;

Solution( graph, nodes );

It fails for all inputs greater than 50kb, presumably inputs with a large set of nodes and edges (ie 50,000 nodes and 40,000 edges). Failing to identify or conceive a faster, more memory efficient algorithm, I'm at a total loss as to what to try next. Thought about making the DFS iterative, but I'm thinking the memory consumption of memoizing thousands of arrays will dwarf that, which seems to be the main problem. I get "Abort Called" and "Runtime Error" on Hackerrank for the 11 tests that fail (as oppose to "Timeout"). Also tried "bitSets" with "union", but the memory consumption turned out to be worse since the bitSets arrays need to be large enough to store numbers up to 50,000.

Constraints:

1 ≤ n,m ≤ 5×10^4
1 ≤ a(i),b(i) ≤ n and a(i) ≠ b(i)
It is guaranteed that graph G does not contain cycles.

Just want to make it clear that I won't get any points for passing all tests since this challenge is locked, this is for educational purposes, mainly on optimization. I'm aware of related SO posts that point to topological sort, but as far as I understand, topological sort will still over count on cases like the one described above, thus not a viable solution. If I misunderstood, please enlighten me. Thank you in advance for your time.

Question: How can I optimize this further? Is there a more efficient approach?

Answer 1

Depth-First Search (DFS) is one good way of solving this problem. Another way would be Breadth-First Search (BFS) which can also run in parallel and can be optimized very well - but all at the cost of much higher code complexity. So my recommendation would be to stick to DFS.

First I have to apologize, but my JavaScript skills are not very good (ie they are non existent) so my solutions below are using Java but the ideas should be easy to port.

Your initial question is missing one very important detail: We only need to find all nodes where the number of reachable nodes is larger or equal than |V| / 2 |V| / 2

Why does that matter? Computing the number of reachable nodes for each node is expensive as we have to do a DFS or BFS starting from every node in the graph. But if we only need to find nodes with the above property, that is much easier.

Let successors(n) be all nodes reachable from n and ancestor(n) be all nodes that can reach n . We can use the following observations to drastically reduce the search space:

• if the number of nodes reachable from n is smaller than |V| / 2 then no node in successors(n) can have a larger number
• if the number of nodes reachable from n is greater or equal than |V| / 2 then all nodes in ancestors(n) will have a larger number

How can we use that?

1. When creating your graph, also create the transposed graph. That means when storing an edge a->b, you store b->a in the transposed graph.
2. Create an array that stores which nodes to ignore, initialize it with false
3. Implement a DFS based function that determines whether a given node has a number of reachable nodes >= |V| / 2 >= |V| / 2 (see below)
4. In that function, ignore nodes that are marked as ignored
5. If for node n the number of nodes is smaller than |V| / 2 , mark all nodes in successors(n) as ignored
6. Else count all nodes in ancestors(n) and mark them as ignored

Solution using Iterative DFS

public int countReachable(int root, boolean[] visited, boolean[] ignored, Graph graph) {
    if (ignored[root]) {
        return 0;
    }

    Stack<Integer> stack = new Stack<>();
    stack.push(root);

    int count = 0;
    while (stack.empty() == false) {
        int node = stack.pop();
        if (visited[node] == false) {
            count++;
            visited[node] = true;
            for (int neighbor : graph.getNeighbors(node)) {
                if (visited[neighbor] == false) {
                    stack.push(neighbor);
                }
            }
        }
    }
    if (count * 2 >= graph.numNodes()) {
        return markAndCountAncestors(root, visited, ignored, graph);
    } else {
        return markSuccessors(root, visited, ignored, graph);
    }
}

Function to mark the Ancestors

This is just another DFS but using the transposed graph. Note that we can reuse the visited array as all values that we will use are false since this is an acyclic graph.

public int markAndCountAncestors(int root, boolean[] visited, boolean[] ignored, Graph graph) {   
    Stack<Integer> stack = new Stack<>();
    stack.push(root);
    visited[root] = false;

    int count = 0;
    while (stack.empty() == false) {
        int node = stack.pop();
        if (visited[node] == false && ignored[node] == false) {
            count++;
            visited[node] = true;
            ignored[node] = true;
            for (int neighbor : graph.transposed.getNeighbors(node)) {
                if (visited[neighbor] == false && ignored[node] == false) {
                    stack.push(neighbor);
                }
            }
        }
    }
    return count;
}

Function to mark the Successors

Note that we already have the successors since they are just the nodes where we set visited to true.

public int markSuccessors(int root, boolean[] visited, boolean[] ignored, Graph graph) {
    for(int node = 0; node < graph.numNodes(); node++) {
        if (visited[node)) {
            ignored[node] = true;
        }
    }
    return 0;
}

Function to compute the result

public void solve(Graph graph) {
    int count = 0;
    boolean[] visited = new boolean[graph.numNodes()];
    boolean[] ignored = new boolean[graph.numNodes()];
    for (int node = 0; node < graph.numNodes(); node++) {
        Arrays.fill(visited, false); // reset visited array
        count += countReachable(node, visited, ignored, graph);
    }
    System.out.println("Result: " + count);
}

On the large test-case you posted, this runs in 7.5 seconds for me. If you invert the iteration order (ie in solve you start with the largest node id) it goes down to 4 seconds, but that somewhat feels like cheating ^^