在python中優化字符串搜索

Question

我必須編寫一個python程序，該程序給出了一個大的50 MB DNA序列和一個較小的序列（約15個字符），返回了所有15個字符的序列的列表，這些序列按它們與給定序列的接近程度以及在何處排序在更大的一個。

我當前的方法是首先獲取所有子序列：

def get_subsequences_of_size(size, data):
    sequences = {}
    i = 0
    while(i+size <= len(data)):
        sequence = data[i:i+size]
        if sequence not in sequences:
            sequences[sequence] = data.count(sequence)
        i += 1
    return sequences

然后根據問題的要求將它們打包在詞典列表中（我忘了獲得職位）：

def find_similar_sequences(seq, data):
    similar_sequences = {}
    sequences = get_subsequences_of_size(len(seq), data)
    for sequence in sequences.keys():
        diffs, muts = calculate_similarity(seq,sequence)
        if diffs not in similar_sequences:
            similar_sequences[diffs] = [{"Sequence": sequence, "Mutations": muts}]
        else:
            similar_sequences[diffs].append({"Sequence": sequence, "Mutations": muts})
        #similar_sequences[sequence] = {"Similarity": (len(sequence)-diffs), "Differences": diffs, "Mutatations": muts}
    return similar_sequences

我的問題是這種運行方式太慢。 使用50MB的輸入，需要30分鍾以上才能完成處理。

Answer 1

那么以下方法呢？

在長序列和每個子序列上使用長度為15的滑動窗口：

• 將開始位置存儲在長序列上
• 計算並存儲相似度

import re
from itertools import islice
from collections import defaultdict

short_seq = 'TGGCGACGGACTTCA'
long_seq = 'AGAACGTTTCGCGTCAGCCCGGAAGTGGTCAGTCGCCTGAGTCCGAACAAAAATGACAACAACGTTTATGACAGAACATT' +\
           'CCTTGCTGGCAACTACCTGAAAATCGGCTGGCCGTCAGTCAATATCATGTCCTCATCAGATTATAAATGCGTGGCGCTGA' +\
           'CGGATTATGACCGTTTTCCGGAAGATATTGATGGCGAGGGGGATGCCTTCTCTCTTGCCTCAAAACGTACCACCACATTT' +\
           'ATGTCCAGTGGTATGACGCTGGTGGAGAGTTCCCCCGGCAGGGATGTGAAGGATGTGAAATGGCGACGGACTTCACCGCA' +\
           'TGAGGCTCCACCAACCACGGGGATACTGTCGCTCTATAACCGTGGCGATCGCCGTCGCTGGTACTGGCCCTGTCCACACT' +\
           'GTGGTGAGTATTTTCAGCCCTGCGGCGATGTGGTTGCTGGTTTCCGTGATATTGCCGATCCCGTGCTGGCAAGTGAGGCG' +\
           'GCTTATATTCAGTGTCCTTCTGGCGACGGACTTCACGCGTCAGCCCGGAAGTGGTCAGTCGCCTGAGTCCGAACAAAAAT'


def window(seq, n=2):
    "Returns a sliding window (of width n) over data from the iterable"
    "   s -> (s0,s1,...s[n-1]), (s1,s2,...,sn), ...                   "
    # from https://docs.python.org/release/2.3.5/lib/itertools-example.html
    it = iter(seq)
    result = tuple(islice(it, n))
    if len(result) == n:
        yield ''.join(result)
    for elem in it:
        result = result[1:] + (elem,)
        yield ''.join(result)

def hamming_distance(s1, s2):
    if len(s1) != len(s2):
        raise ValueError("Undefined for sequences of unequal length")
    return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))

k = len(short_seq)
locations = defaultdict(list)
similarities = defaultdict(set)

for start, subseq in enumerate(window(long_seq, k)):
    locations[subseq].append(start)
    similarity = hamming_distance(subseq, short_seq) # substitute with your own similarity function
    similarities[similarity].add(subseq)

with open(r'stack46268997.txt', 'w') as f:
    for similarity in sorted(similarities.keys()):
        f.write("Sequence(s) which differ in {} base(s) from the short sequence:\n".format(similarity))
        for subseq in similarities[similarity]:
            f.write("{} at location(s) {}\n".format(subseq, ', '.join(map(str, locations[subseq]))))
        f.write('\n')

這將輸出子序列列表，這些子序列按它們與給定序列的接近程度排序。

Sequence(s) which differ in 0 base(s) from the short sequence:
TGGCGACGGACTTCA at location(s) 300, 500

Sequence(s) which differ in 5 base(s) from the short sequence:
TGGCGATCGCCGTCG at location(s) 362

Sequence(s) which differ in 6 base(s) from the short sequence:
TGGCAACTACCTGAA at location(s) 86
TGGTGAGTATTTTCA at location(s) 401
TGGCGAGGGGGATGC at location(s) 191

Sequence(s) which differ in 7 base(s) from the short sequence:
ATGTGAAGGATGTGA at location(s) 283
AGGGGGATGCCTTCT at location(s) 196
TGACAACAACGTTTA at location(s) 53
CGCTGACGGATTATG at location(s) 154
TTATGACCGTTTTCC at location(s) 164
TGGTTGCTGGTTTCC at location(s) 430
TCGCGTCAGCCCGGA at location(s) 8
AGTCGCCTGAGTCCG at location(s) 30, 536
CGGCGATGTGGTTGC at location(s) 422

[... and so on...]

我還在50 MB FASTA文件上運行了該腳本。 在我的機器上，這需要42秒鍾來計算結果，而又需要30秒鍾才能將結果寫到文件中（打印出來將花費更長的時間！）