通过数据特征是否相交对数据进行分组

Question

Suppose we have a table:

id  | aliases
-------------
0   | ['a0', 'a1', 'a4', 'a11']
1   | ['a3', 'a5']
2   | ['a16', 'a18']
3   | ['a6', 'a8', 'a10']
4   | ['a7', 'a8', 'a9']
5   | ['a3', 'a12', 'a14']
6   | ['a5', 'a16', 'a17']

and I'd like to group all id 's together that map to the same aliases ; in other words, the end result groups together all id 's that have aliases that intersect, applied recursively. In the above case, we would have:

• 0 maps to ['a0', 'a1', 'a4', 'a11']
• 1 , 2 , 5 , and 6 map to ['a3', 'a5', 'a12', 'a14', 'a16', 'a17', 'a18']
• 3 and 4 map to ['a6', 'a7', 'a8', 'a9', 'a10']

Is there an efficient way to do this? In my actual use case I have around 15M rows.

There is a naïve approach of streaming the rows and checking if each element of aliases in each new row is in the aliases processed so far; if so, collecting the id 's together of all rows with matching aliases , and mapping them to the union of the aliases matched.

However, this approach seems computationally impractical.

Answer 1

To run a O(n*len(groupcount) complexity code on this table shouldn't be that hard, just off the top of my mind:

Assume you have id as a list of id and aliases as list of lists, you can do:

bins = []
sets = []
for i in id: # Assume from (0 - n)
    alias = aliases[i]
    in_set = False
    for j in range(len(sets)):
        if len(sets[j].intersection(set(alias))) > 0:
            sets[j].update(set(alias)) # add alias to set, if any difference
            in_set = True
            bins[j].append(i) # append id to bins
            break
    if not in_set:
        bins.append([i])
        sets.append(set(alias))

bins will contain the id groups, and corresponding element in sets will contain the alias groups, you can use list() to convert these sets back to list . And since all set operations are hash based this ensure your program runs in O(n*groupcount) time.