查找两个列表中不包含公共字符的所有字符串对

Question

I have two lists of strings, and wish to find all pairs of strings between them that contain no common characters. eg

list1 = ['abc', 'cde']
list2 = ['aij', 'xyz', 'abc']

desired output = [('abc', 'xyz'), ('cde', 'aij'), ('cde', 'xyz')]

I need this to be as efficient as possible because I am processing lists with millions of strings. At the moment, my code follows this general pattern:

output = []

for str1 in list1:    
    for str2 in list2:
        if len(set(str1) & set(str2)) == 0: 
             output.append((str1, str2))

This is O(n^2) and is taking many hours to run, does anyone have some suggestions for how to speed this up? Perhaps there is a way to take advantage of characters in each string being sorted?

Thank you very much in advance!

Answer 1

Here's another tack, focusing on lowering the set operations to bit twiddling and combining words that represent the same set of letters:

import collections
import string


def build_index(words):
    index = collections.defaultdict(list)
    for word in words:
        chi = sum(1 << string.ascii_lowercase.index(letter) for letter in set(word))
        index[chi].append(word)
    return index


def disjoint_pairs(words1, words2):
    index1 = build_index(words1)
    index2 = build_index(words2)
    for chi1, words1 in index1.items():
        for chi2, words2 in index2.items():
            if chi1 & chi2:
                continue
            for word1 in words1:
                for word2 in words2:
                    yield word1, word2


print(list(disjoint_pairs(["abc", "cde"], ["aij", "xyz", "abc"])))

Answer 2

Try this and tell me if there is any improvement:

import itertools

[i for i in itertools.product(list1, list2) if len(i[0]+i[1])==len(set(i[0]+i[1]))]

Output:

[('abc', 'xyz'), ('cde', 'aij'), ('cde', 'xyz')]

Answer 3

It's tricky to analyze the running time of this algorithm, but it's what I'd try first. The idea is that, given a letter, we can split the problem into three subproblems: (words without the letter, words without the letter), (words without the letter, words with the letter), (words with the letter, words without the letter). The code below chooses this letter (the "pivot") to maximize the number of pairs eliminated. In the base case, no pair can be eliminated, and we just output all pairs.

Python 3, optimized for readability over running time.

import collections


def frequencies(words):
    return collections.Counter(letter for word in words for letter in set(word))


def partition(pivot, words):
    return (
        [word for word in words if pivot not in word],
        [word for word in words if pivot in word],
    )


def disjoint_pairs(words1, words2):
    freq1 = frequencies(words1)
    freq2 = frequencies(words2)
    pivots = set(freq1.keys()) & set(freq2.keys())
    if pivots:
        pivot = max(pivots, key=lambda letter: freq1[letter] * freq2[letter])
        no1, yes1 = partition(pivot, words1)
        no2, yes2 = partition(pivot, words2)
        yield from disjoint_pairs(no1, no2)
        yield from disjoint_pairs(no1, yes2)
        yield from disjoint_pairs(yes1, no2)
    else:
        for word1 in words1:
            for word2 in words2:
                yield (word1, word2)


print(list(disjoint_pairs(["abc", "cde"], ["aij", "xyz", "abc"])))

Answer 4

You can use recursion with a generator:

from functools import reduce
list1 = ['abc', 'cde']
list2 = ['aij', 'xyz', 'abc']
def pairs(d, c = []):
   if not d and not reduce(lambda x, y:set(x)&set(y), c):
      yield tuple(c)
   elif d:
      yield from [i for k in d[0] for i in pairs(d[1:], c+[k])]

print(list(pairs([list1, list2])))

Output:

[('abc', 'xyz'), ('cde', 'aij'), ('cde', 'xyz')]

This answer uses functools.reduce in order to handle cases where the number of input lists is greater than two. That way, the set intersection of all the elements in the potential sublist can more easily computed.

Answer 5

I was thinking about how to exploit the fact that the strings are ordered and came up with the following crude idea:

Step 1 : Sort the second list, only with respect to the first character of the strings:

from itertools import product

list1 = ['abc', 'cde']
list2 = ['aij', 'xyz', 'abc']
list2 = sorted(list2, key=(lambda s: s[0]))

Step 2 : Find for each character c in the alphabet the index of the first element s in list2 which has a first character s[0] larger than c . (Here I assume that all the elements in the strings are actually characters from the alphabet and all lower case!)

alphabet = 'abcdefghijklmnopqrstuvwxyz'
bounds = {}
for c in alphabet:
    bounds[c] = len(list2)
    for i, s in enumerate(list2):
        if c < s[0]:
            bounds[c] = i
            break

Step 3 : With that preparation the iteration over the two lists can be optimised a little bit. For an element str1 from list1 you most likely don't have to go through all elements in list2 and do the check: At a certain point you know that the rest of the strings in list2 are distinct to str1 .

output = []
for str1 in list1:
    last_char = str1[-1]
    for str2 in list2[:bounds[last_char]]:
        if len(set(str1) & set(str2)) == 0:
            output.append((str1, str2))
    output += [*product([str1], list2[bounds[last_char]:])]

Caveats: I don't know if that actually helps! You have to sort list2 and still have to collect the certain combinations (this part: output += [*product([str1], list2[bounds[last_char]:])] ), and I don't know how much that costs.

PS: The code here is only intended to illustrate the idea (and I hope it doesn't contain any errors - it's late here), an actual implementation would look differently.