提高 Python 3 中滑动窗口片段函数的性能

Question

I have a script in Python 3.6.8 which reads through a very large text file, where each line is an ASCII string drawn from the alphabet {a,b,c,d,e,f} .

For each line, I have a function which fragments the string using a sliding window of size k , and then increments a fragment counter dictionary fragment_dict by 1 for each fragment seen.

The same fragment_dict is used for the entire file, and it is initialized for all possible 5^k fragments mapping to zero.

I also ignore any fragment which has the character c in it. Note that c is uncommon, and most lines will not contain it at all.

def fragment_string(mystr, fragment_dict, k):
    for i in range(len(mystr) - k + 1):

        fragment = mystr[i:i+k]
        if 'c' in fragment:
            continue

        fragment_dict[fragment] += 1

Because my file is so large, I would like to optimize the performance of the above function as much as possible. Could anyone provide any potential optimizations to make this function faster?

I'm worried I may be rate limited by the speed of Python loops, in which case I would need to consider dropping down into C/Cython.

Answer 1

Numpy may help in speeding up your code:

x = np.array([ord(c) - ord('a') for c in mystr])
filter = np.geomspace(1, 5**(k-1), k, dtype=int)
fragment_dict = collections.Counter(np.convolve(x, filter,mode='valid'))

The idea is, represent each k length segment is a k-digit 5-ary number. Then, converting a list of 0-5 integers equivalent to the string to its 5-ary representation is equivalent to applying a convolution with [1,5,25,125,...] as filter.