使用python快速查找大字典中的正则表达式匹配

Question

I have a large dictionary containing regex values as the key and a numeric value as a value, and given a corpus (broken down into a list of individual word tokens) I would like to to find the regex value that best matches my word to obtain its respective value.

The dictionary contains many regex values that are ambiguous, in the sense that a word may have multiple regex matches, and therefore you would want to find the longest regex or 'best match' (ex: dictionary contains affect+, as well as affected an affection)

My issue is when running a large text sample through the dictionary and finding the regex match of each word token, it takes a long amount of time (0.1s per word), which obviously adds up over 1000's of words. This is because it goes through the whole dictionary each time to find the 'best match'.

Is there a faster way to achieve this? Please see the problematic part of my code below.

for word in textTokens:
    for reg,value in dictionary.items():
        if(re.match(reg, word)):
            matchedWords.append(reg)

Answer 1

Because you mentioned the input regexes have the structure of word+ a simple word and a plus regex symbol, you can use a modified version of the optimal Aho-Corasick algorithm , in this algorithm you make a finite-state-machine from in search patterns that can easily be modified accept some regex signs easily, like in your case a very simplistic solution would be to pad your keys to length of longest words in the list and accept anything that comes after the padding, there is an easy to implement wildcards are '.' and '?', for * one have to go to end of the word or return and follow the other path through list, which can be exponential number of choices in constant memory(any and all deterministic finite automatons).

A finite state machine for a list of your regex keys can be made in linear time, meaning it takes time proportional to the sum of length of your dictionary keys. as explained here there is also support for longest matched word in the dictionary.

Answer 2

Thank you everyone for your answers, they were all very interesting and helpful. Ultimately i was forced to hand in my project without implementing a clean solution due to time constraints, but i have finally been able togo back and implement the Trie structure recommended by Stef . With a trie implemented, i was able to increase the speed of my algorithm by over 30x.

For anyone interested in seeing my final project, the application can be found here , and the code on my Github