找到不匹配模式的效率

Question

我正在研究一個簡單的生物信息學問題。 我有一個有效的解決方案，但這是非常低效的。 我怎樣才能提高效率？

---

問題：

在字符串g找到長度為k模式，假設k mer最多可能有d不匹配。

這些字符串和模式都是基因組的 - 所以我們的可能字符集是{A, T, C, G} 。

我將調用函數FrequentWordsMismatch(g, k, d) 。

所以，這里有一些有用的例子：

FrequentWordsMismatch('AAAAAAAAAA', 2, 1) → ['AA', 'CA', 'GA', 'TA', 'AC', 'AG', 'AT']

這是一個更長的例子，如果你實現這個並想測試：

FrequentWordsMisMatch('CACAGTAGGCGCCGGCACACACAGCCCCGGGCCCCGGGCCGCCCCGGGCCGGCGGCCGCCGGCGCCGGCACACCGGCACAGCCGTACCGGCACAGTAGTACCGGCCGGCCGGCACACCGGCACACCGGGTACACACCGGGGCGCACACACAGGCGGGCGCCGGGCCCCGGGCCGTACCGGGCCGCCGGCGGCCCACAGGCGCCGGCACAGTACCGGCACACACAGTAGCCCACACACAGGCGGGCGGTAGCCGGCGCACACACACACAGTAGGCGCACAGCCGCCCACACACACCGGCCGGCCGGCACAGGCGGGCGGGCGCACACACACCGGCACAGTAGTAGGCGGCCGGCGCACAGCC', 10, 2) → ['GCACACAGAC', 'GCGCACACAC']

憑借我天真的解決方案，第二個例子很容易花費大約60秒，盡管第一個例子非常快。

---

天真的解決方案：

我的想法是，對於g中的每個k長度段，找到每個可能的“鄰居”（例如，具有多達d個不匹配的其他k長度段）並將這些鄰居添加為字典的鍵。 然后我計算每個鄰居kmers出現在字符串g中的次數 ，並將其記錄在字典中。

顯然，這是一種有點糟糕的方式，因為鄰居的數量像k和d一樣瘋狂地擴展，並且必須通過每個鄰居掃描字符串使得這種實現非常緩慢。 但是，這就是我要求幫助的原因。

我會把我的代碼放在下面。 打開包裝肯定會有很多新手錯誤，所以感謝您的時間和精力。

def FrequentWordsMismatch(g, k, d):
    '''
    Finds the most frequent k-mer patterns in the string g, given that those 
    patterns can mismatch amongst themselves up to d times

    g (String): Collection of {A, T, C, G} characters
    k (int): Length of desired pattern
    d (int): Number of allowed mismatches
    '''
    counts = {}
    answer = []

    for i in range(len(g) - k + 1):
        kmer = g[i:i+k]
        for neighborkmer in Neighbors(kmer, d):
            counts[neighborkmer] = Count(neighborkmer, g, d)

    maxVal = max(counts.values())

    for key in counts.keys():
        if counts[key] == maxVal:
            answer.append(key)

    return(answer)


def Neighbors(pattern, d):
    '''
    Find all strings with at most d mismatches to the given pattern

    pattern (String): Original pattern of characters
    d (int): Number of allowed mismatches
    '''
    if d == 0:
        return [pattern]

    if len(pattern) == 1:
        return ['A', 'C', 'G', 'T']

    answer = []

    suffixNeighbors = Neighbors(pattern[1:], d)

    for text in suffixNeighbors:
        if HammingDistance(pattern[1:], text) < d:
            for n in ['A', 'C', 'G', 'T']:
                answer.append(n + text)
        else:
            answer.append(pattern[0] + text)

    return(answer)


def HammingDistance(p, q):
    '''
    Find the hamming distance between two strings

    p (String): String to be compared to q
    q (String): String to be compared to p
    '''
    ham = 0 + abs(len(p)-len(q))

    for i in range(min(len(p), len(q))):
        if p[i] != q[i]:
            ham += 1

    return(ham)


def Count(pattern, g, d):
    '''
    Count the number of times that the pattern occurs in the string g, 
    allowing for up to d mismatches

    pattern (String): Pattern of characters
    g (String): String in which we're looking for pattern
    d (int): Number of allowed mismatches
    '''
    return len(MatchWithMismatch(pattern, g, d))

def MatchWithMismatch(pattern, g, d):
    '''
    Find the indicies at which the pattern occurs in the string g, 
    allowing for up to d mismatches

    pattern (String): Pattern of characters
    g (String): String in which we're looking for pattern
    d (int): Number of allowed mismatches
    '''
    answer = []
    for i in range(len(g) - len(pattern) + 1):
        if(HammingDistance(g[i:i+len(pattern)], pattern) <= d):
            answer.append(i)
    return(answer)

---

更多測試

FrequentWordsMismatch('ACGTTGCATGTCGCATGATGCATGAGAGCT', 4, 1) → ['ATGC', 'ATGT', 'GATG']

FrequentWordsMismatch('AGTCAGTC', 4, 2) → ['TCTC', 'CGGC', 'AAGC', 'TGTG', 'GGCC', 'AGGT', 'ATCC', 'ACTG', 'ACAC', 'AGAG', 'ATTA', 'TGAC', 'AATT', 'CGTT', 'GTTC', 'GGTA', 'AGCA', 'CATC']

FrequentWordsMismatch('AATTAATTGGTAGGTAGGTA', 4, 0) → ["GGTA"]

FrequentWordsMismatch('ATA', 3, 1) → ['GTA', 'ACA', 'AAA', 'ATC', 'ATA', 'AGA', 'ATT', 'CTA', 'TTA', 'ATG']

FrequentWordsMismatch('AAT', 3, 0) → ['AAT']

FrequentWordsMismatch('TAGCG', 2, 1)  → ['GG', 'TG']

Answer 1

單獨討論您的問題描述而不是您的示例（由於我在評論中解釋的原因），一種方法是：

s = "CACAGTAGGCGCCGGCACACACAGCCCCGGGCCCCGGGCCGCCCCGGGCCGGCGGCCGCCGGCGCCGGCACACCGGCACAGC"\
    "CGTACCGGCACAGTAGTACCGGCCGGCCGGCACACCGGCACACCGGGTACACACCGGGGCGCACACACAGGCGGGCGCCGGG"\
    "CCCCGGGCCGTACCGGGCCGCCGGCGGCCCACAGGCGCCGGCACAGTACCGGCACACACAGTAGCCCACACACAGGCGGGCG"\
    "GTAGCCGGCGCACACACACACAGTAGGCGCACAGCCGCCCACACACACCGGCCGGCCGGCACAGGCGGGCGGGCGCACACAC"\
    "ACCGGCACAGTAGTAGGCGGCCGGCGCACAGCC"

def frequent_words_mismatch(g,k,d):
    def num_misspellings(x,y):
        return sum(xx != yy for (xx,yy) in zip(x,y))

    seen = set()
    for i in range(len(g)-k+1):
        seen.add(g[i:i+k])

    # For each unique sequence, add a (key,bin) pair to the bins dictionary
    #  (The bin is initialized to a list containing only the sequence, for now)
    bins = {seq:[seq,] for seq in seen}
    # Loop again through the unique sequences...
    for seq in seen:
        # Try to fit it in *all* already-existing bins (based on bin key)
        for bk in bins:
            # Don't re-add seq to it's own bin
            if bk == seq: continue
            # Test bin keys, try to find all appropriate bins
            if num_misspellings(seq, bk) <= d:
                bins[bk].append(seq)

    # Get a list of the bin keys (one for each unique sequence) sorted in order of the
    #   number of elements in the corresponding bins
    sorted_keys = sorted(bins, key= lambda k:len(bins[k]), reverse=True)

    # largest_bin_key will be the key of the largest bin (there may be ties, so in fact
    #   this is *a* key of *one of the bins with the largest length*).  That is, it'll
    #   be the sequence (found in the string) that the most other sequences (also found
    #   in the string) are at most d-distance from.
    largest_bin_key = sorted_keys[0]

    # You can return this bin, as your question description (but not examples) indicate:
    return bins[largest_bin_key]

largest_bin = frequent_words_mismatch(s,10,2)
print(len(largest_bin))     # 13
print(largest_bin)

（這個）最大的bin包含：

['CGGCCGCCGG', 'GGGCCGGCGG', 'CGGCCGGCGC', 'AGGCGGCCGG', 'CAGGCGCCGG',
 'CGGCCGGCCG', 'CGGTAGCCGG', 'CGGCGGCCGC', 'CGGGCGCCGG', 'CCGGCGCCGG',
 'CGGGCCCCGG', 'CCGCCGGCGG', 'GGGCCGCCGG']

它是O （n ** 2），其中n是唯一序列的數量，並在我的計算機上在大約0.1秒內完成。

Answer 2

問題描述在幾個方面是模棱兩可的，所以我將通過這些例子。 您似乎想要字母表中的所有k長度字符串(A, C, G, T} ，使得與g連續子串的匹配數最大 - 其中“匹配”表示最多與字符相等d字符不等式。

我忽略了你的HammingDistance()函數即使在輸入長度不同的情況下也會產生一些東西，主要是因為它對我沒有多大意義;-)，但部分是因為不需要得到你想要的結果你提供的任何例子。

下面的代碼生成了所有示例中所需的結果，從而產生了您給出的輸出列表的排列。 如果你想要規范輸出，我建議在返回之前對輸出列表進行排序。

該算法非常簡單，但依賴於itertools來“以C速度”進行重組合提升。 所有的例子都在第二個總數下運行良好。

對於g每個長度k連續子串，考慮d不同索引位置的所有combinations(k, d)組。 有4**d方法用來自{A, C, G, T}字母填充那些索引位置，並且每種方式都是“一種模式”，其匹配具有最多d差異的子字符串。 通過記住已生成的模式來清除重復項; 這比制作英雄的努力更快地生成只有獨特的模式開始。

所以，總的來說，時間要求是O(len(g) * k**d * 4**d) = O(len(g) * (4*k)**d ，其中k**d是，對於k和d相當小的值，二項式系數combinations(k, d)的誇大其詞。重要的是要注意的是 - 毫不奇怪 - 它在d是指數的。

def fwm(g, k, d):
    from itertools import product, combinations
    from collections import defaultdict

    all_subs = list(product("ACGT", repeat=d))
    all_ixs = list(combinations(range(k), d))
    patcount = defaultdict(int)

    for starti in range(len(g)):
        base = g[starti : starti + k]
        if len(base) < k:
            break
        patcount[base] += 1
        seen = set([base])
        basea = list(base)
        for ixs in all_ixs:
            saved = [basea[i] for i in ixs]
            for newchars in all_subs:
                for i, newchar in zip(ixs, newchars):
                    basea[i] = newchar
                candidate = "".join(basea)
                if candidate not in seen:
                    seen.add(candidate)
                    patcount[candidate] += 1
            for i, ch in zip(ixs, saved):
                basea[i] = ch

    maxcount = max(patcount.values())
    return [p for p, c in patcount.items() if c == maxcount]

編輯：獨特地生成模式

而不是通過保留迄今為止看到的那些副本來清除重復項，它足以直接防止生成重復項。 事實上，下面的代碼更短更簡單，雖然有點微妙。 作為少量冗余工作的回報，有一些對inner()函數的遞歸調用。 哪種方式更快似乎取決於具體的輸入。

def fwm(g, k, d):
    from collections import defaultdict

    patcount = defaultdict(int)
    alphabet = "ACGT"
    allbut = {ch: tuple(c for c in alphabet if c != ch)
              for ch in alphabet}

    def inner(i, rd):
        if not rd or i == k:
            patcount["".join(base)] += 1
            return
        inner(i+1, rd)
        orig = base[i]
        for base[i] in allbut[orig]:
            inner(i+1, rd-1)
        base[i] = orig

    for i in range(len(g) - k + 1):
        base = list(g[i : i + k])
        inner(0, d)

    maxcount = max(patcount.values())
    return [p for p, c in patcount.items() if c == maxcount]