Python - 遍历边列表，对于具有特定属性的节点，找到具有不同特定属性的所有连接节点？

Question

I have an edge list containing 24 000 different edges between produts. An edge is created between A and B if product B is a sub component of A.

The edge list is on the following format:

 Parent | Child | Root | Child Meta
  AA1      BB1    AA1      ...  
  AA1      BB2    AA1      ...
  BB2      CC1    AA1      ...  
  AA2      BB3    AA2
  AA2      BB4    AA2
  BB4      CC1    AA2      ... 
  BB4      DD1    AA2      ...
  DD1      EE1    AA2
  DD1      EE2    AA2
  BB4      FF1    AA2
  FF1      GG1    AA2      ...
  GG1      EE3    AA2

So by grouping by Root I want, for all parents on the form DD* and FF* , find children on the form EE* they have a direct connection with. In the example above I want the output dataframe to look like

 Parent | Child | Root | Child Meta
   DD1     EE1    AA2      ... 
   DD1     EE2    AA2      ...
   FF1     EE3    AA2      ...

The only way I know how to do this is by iterating over a pandas DataFrame and using recursive functions iterating over the children until I hit an EE* child. This takes forever. Is there a smart way to use networkx here maybe? Or are there any other way I can do this using pandas that would be faster?

Answer 1

If I understand the issue correctly, then it might be faster if you start at the bottom and find nodes going upwards.

Since you know the subset of children (E*) you want to find, if you start with the target children, all parents are by definition part of the result, and you don't have to filter.

In a plain iterative Python approach, something like this would find all parent nodes for "E*" children:

(Please note that I have added an extra line with "BB3 DD1 AA2" to have another duplicate.)

data = """AA1      BB1    AA1
  AA1      BB2    AA1 
  BB2      CC1    AA1 
  AA2      BB3    AA2
  AA2      BB4    AA2
  BB4      CC1    AA2 
  BB3      DD1    AA2
  BB4      DD1    AA2
  DD1      EE1    AA2
  DD1      EE2    AA2
  BB4      FF1    AA2
  FF1      GG1    AA2
  GG1      EE3    AA2"""

# tuple (parent, child, root)
tuples = {tuple(l.split()) for l in data.split("\n")}

parentsByChild = {}
for node in tuples:
    p = set(parentsByChild.get(node[1], frozenset()))
    p.add(node)
    parentsByChild[node[1]] = frozenset(p)
# alternatively:
# from itertools import groupby
# parentsByChild = {c:frozenset(nodes) for c, nodes in groupby(sorted(tuples, key=lambda n: n[1]), lambda n: n[1])}

def expand(nodes):
    todo, found = set(nodes), set() 
    while todo:
        node = todo.pop()        
        if not node in found:
            found.add(node)
            todo.update((p for p in parentsByChild.get(node[0], set()) if p not in found))
    return found

leaves = {n for n in tuples if n[1].startswith("E")}
for t in expand(leaves):
    print(t)

This should be linear in the number of edges: We iterate over them once to collect the tuples and a second time to group the parents. The expand call iterates over all "interesting" children and their parents, expanding parents only for new nodes, so we never do work twice for the same node.