从大字典中匹配子字符串的最快方法

Question

I have some (usually < 300 symbols length) string like 'aabbccdcabcbbacdaaa'.

There is python dictionary where keys are strings in similar format, eg 'bcccd', key length varies from 10 to 100 symbols. The dictionary has half a million items.

I need to match my initial string with dictionary's value or find out that there are no proper values in dictionary. Matching condition: dictionary key should be somewhere within string (strict matching).

What is the best way, in terms of computational speed, to do it? I feel there should be some tricky way to hash my initial string and dictionary keys as to apply some clever ways of substring search (like Rabin-Karp or Knuth-Morris-Pratt). Or suffix tree-like structure could be a good solution?

Answer 1

Just found a reasonable implementation of Aho-Corasick for Python - pyahocorasick . Taking from the example at the end of the page:

import ahocorasick
A = ahocorasick.Automaton()

for k, v in your_big_dict.iteritems():
    A.add_word(k, v)

A.make_automaton()
for item in A.iter(your_long_string):
    print(item)

Answer 2

You can use the following format:

for key in your_dictionary:
    if key in your_string:
        print(key+' is in both your string and the dictionary. It has the value '+str(your_dictionary[key]))

If you want this changed in any way, let me know in the comments, I'll be happy to update.

Answer 3

def search(string, dict_search):
    # If those 2 lines are too expensive, calculate them and pass as arguments
    max_key = max(len(x) for x in dict_search)
    min_key = min(len(x) for x in dict_search)

    return set(
        string[x:x+i] 
        for i in range(min_key, max_key+1)
        for x in range(len(string)-i+1)
        if string[x:x+i] in dict_search
    )

Running:

>>> search('aabbccdcabcbbacdaaa', {'aaa', 'acd', 'adb', 'bccd', 'cbbb', 'abc'})
{'aaa', 'abc', 'acd', 'bccd'}