设置vs. Regex进行字符串查找，哪个更具扩展性？

Question

Suppose that I need to handle a very big list of words, and I need to count the number of times I find any of those words in a piece of text I have. Which is the best option in terms of scalability?

Option I (regex)

>>> import re
>>> s = re.compile("|".join(big_list))
>>> len(s.find_all(sentence))

Option II (sets)

>>> s = set(big_list)
>>> len([word for word in sentence.split(" ") if word in s]) # O(1) avg lookup time

Example: if the list is ["cat","dog","knee"] and the text is "the dog jumped over the cat, but the dog broke his knee" the final result should be: 4

PS Any other option is welcome

Answer 1

If your words are alphanumeric, I might use something like:

s = set(big_list)
sum(1 for x in re.finditer(r'\b\w+\b',sentence) if x.group() in s)

Since the membership test for a set is on average O(1), this algorithm becomes O(N+M) where N is the number of words in the sentence and M is the number of elements in big_list. Not too shabby. It also does pretty well in terms of memory usage.

Answer 2

A scalable method would be sorting the input dictionary and words from the text then doing the matching using two iterators. You can also use a trie for even better performance. I don't know the internal representation of the set, however, using a large regex would be a total overkill.