DFS 查找所有可能的路径很慢

Question

I wrote DFS-like algorithm to find all possible paths starting from zero level. With 2,000 nodes and 5,000 edges, below code execution is extremely slow. Any suggestion for this algorithm?

    all_path = []

    def printAllPathsUntil(s, path):
        path.append(s)
        if s not in adj or len(adj[s]) <= 0:
            all_path.append(path[:]) # EDIT2
        else:
            for i in adj[s]:
                printAllPathsUntil(i, path)
        path.pop()

    for point in points_in_start:
        path = []
        printAllPathsUntil(point, path)

And the adj holds edges; start position as key and destination list as value.

    points_in_start = [0, 3, 7]
    adj = {0: [1, 8],
           1: [2, 5],
           2: [],
           3: [2, 4],
           4: [],
           5: [6],
           6: [],
           7: [6],
           8: [2]
           }

EDIT1

• This is a DAG. No cycles.

Answer 1

The problem with your algorithm is that it will do a lot of repeated work. This is not the case in your example, as the only time when a node is reached by two other nodes, it is a leaf node, like C , but imaging an edge from D to B : That would mean that the entire sub-graph starting at B is visited again! For a graph with 2000 nodes, this will result in a significant slow-down.

To counter this, you can use memoization, but this means that you have to restructure your algorithm to instead of adding to the existing path and then adding that path to all_paths , it has to return the (partial) paths starting at the current node and combine those to the full paths with the parent node. You can then use functools.lru_cache to re-use all those partial results when you visit B again coming from another node.

from functools import lru_cache

@lru_cache(None)
def getAllPathsUntil(s):
    if s not in adj or not adj[s]:
        return [ [s] ]
    else:
        return [ [s, *p] for a in adj[s]
                         for p in getAllPathsUntil(a)]

all_paths = []
for point in points_in_start:
    all_paths.extend(getAllPathsUntil(point))

Answer 2

As pointed out already in the comments and other answers, remembering downstream paths of previously visited nodes is an area of optimization.

Here's my attempt at implementing that.

Here, downstream_paths is a dictionary where we remember, for each visited non-leaf node, the downstream paths from that node.

I have mentioned %%timeit results towards the end for a small test case containing a small case of "revisited non-leafs". Since my test case had only one case of a non-leaf node being re-visited, the improvement was only modest. Perhaps in your large-scale dataset, there will be a wider gap in the performance.

Input data:

points_in_start = [0, 3, 7]
adj = {0: [1, 8],
       1: [2, 5],
       2: [],
       3: [2, 4],
       4: [],
       5: [6],
       6: [],
       7: [6],
       8: [2],     # Non-leaf node "2" is a child of both "8" and "3"
       
       2:[10],
       
       10:[11,18],
       11:[12,15],
       12:[],
       15:[16],
       16:[],
       18:[12]
      }

The modified code:

%%timeit

downstream_paths = {}                                 # Maps each node to its
                                                      # list of downstream paths
                                                      # starting with that node.

def getPathsToLeafsFrom(s):      # Returns list of downstream paths starting from s
                                 # and ending in some leaf node.
    children = adj.get(s, [])
    if not children:                                  # s is a Leaf
        paths_from_s = [[s]]
    else:                                             # s is a Non-leaf
        ds_paths = downstream_paths.get(s, [])        # Check if s was previously visited
        if ds_paths:                                  # If s was previously visited.
            paths_from_s = ds_paths
        else:                                         # s was not visited earlier.
            paths_from_s = []                         # Initialize
            for child in children:
                paths_from_child = getPathsToLeafsFrom(child)   # Recurse for each child
                for p in paths_from_child:
                    paths_from_s.append([s] + p)
            downstream_paths[s] = paths_from_s       # Cache this, to use when s is re-visited
    return paths_from_s

path = []
for point in points_in_start:
    path.extend(getPathsToLeafsFrom(point))

Output:

from pprint import pprint
pprint (all_path)

[[0, 1, 2, 10, 11, 12],
 [0, 1, 2, 10, 11, 15, 16],
 [0, 1, 2, 10, 18, 12],
 [0, 1, 5, 6],
 [0, 8, 2, 10, 11, 12],
 [0, 8, 2, 10, 11, 15, 16],
 [0, 8, 2, 10, 18, 12],
 [3, 2, 10, 11, 12],
 [3, 2, 10, 11, 15, 16],
 [3, 2, 10, 18, 12],
 [3, 4],
 [7, 6]]

Timing results: Original posted code:

10000 loops, best of 3: 63 µs per loop

Timing results: Optimized code:

10000 loops, best of 3: 43.2 µs per loop