拼写检查器加快

Question

i have this spell checker which i have writted:

import operator

class Corrector(object):

    def __init__(self,possibilities):
        self.possibilities = possibilities

    def similar(self,w1,w2):
        w1 = w1[:len(w2)]
        w2 = w2[:len(w1)]
        return sum([1 if i==j else 0 for i,j in zip(w1,w2)])/float(len(w1))

    def correct(self,w):
        corrections = {}
        for c in self.possibilities:
            probability = self.similar(w,c) * self.possibilities[c]/sum(self.possibilities.values())
            corrections[c] = probability
        return max(corrections.iteritems(),key=operator.itemgetter(1))[0]

here possibilities is a dictionary like:

{word1:value1} where value is the number of times the word appeared in the corpus.

The similar function returns the probability of similarity between the words: w1 and w2.

in the correct function, you see that the software loops through all possible outcomes and then computes a probability for each of them being the correct spelling for w.

can i speed up my code by somehow removing the loop?

now i know there might be no answer to this question, if i can't just tell me that i cant!

Answer 1

Here you go....

from operator import itemgetter
from difflib import SequenceMatcher

class Corrector(object):

    def __init__(self, possibilities):
        self.possibilities = possibilities
        self.sums = sum(self.possibilities.values())

    def correct(self, word):
        corrections = {}
        sm = SequenceMatcher(None, word, '')
        for w, t in self.possibilities.iteritems():
            sm.b = w
            corrections[w] = sm.ratio() * t/self.sums
        return max(corrections.iteritems(),key=itemgetter(1))[0]

Answer 2

您可以简单地缓存correct的结果，以便在下一个具有相同世界的通话中，您无需任何计算就可以知道答案。

Answer 3

You typically don't want to check the submitted token against all the tokens in your corpus. The "classic" way to reduce the necessary computations (and thus to reduce the calls in your for loop) is to maintain an index of all the (tri-)grams present in your document collection. Basically, you maintain a list of all the tokens of your collection on the one side, and, on the other side, an hash table which keys are the grams, and which values are the index of the tokens in the list. This can be made persistent with a DBM-like database.

Then, when it comes about checking the spelling of a word, you split it into grams, search for all the tokens in your collection that contain the same grams, sort them by gram similarity with the submitted token, and then , you perform your distance-computations.

Also, some parts of your code could be simplified. For example, this:

def similar(self,w1,w2):
    w1 = w1[:len(w2)]
    w2 = w2[:len(w1)]
    return sum([1 if i==j else 0 for i,j in zip(w1,w2)])/float(len(w1))

can be reduced to:

def similar(self, w1, w2, lenw1):
    return sum(i == j for i, j in zip(w1,w2)) / lenw1

where lenw1 is the pre-computed length of "w1".