比较大型清单中的项目-查找长度相差1个字母的项目-Python

Question

I am looking to solve a problem I've been stuck on in Python. I have a single file, with one column, which contains approx. 6,000 lines. For each line, every item is unique (this file was filtered to remove duplicates froma 40,000 line file). The items in each row vary in length, where some are equal in length to others.

An example of a single line:

IGHV3-30/33rn-IGHJ4-CARDPSLSSMITFGGVIVTRGYFDYW

Or more examples with tab separated after third "-" (differing first parts):

IGHV3-23-IGHJ4  CAKDRGYTGYGVYFDYW
IGHV4-39-IGHJ4  CARHDILTGYSYYFDYW
IGHV3-23-IGHJ3  CAKSGGWYLSDAFDIW
IGHV4-39-IGHJ4  CARTGFGELGFDYW
IGHV1-2-IGHJ2   CARDSDYDWYFDLW
IGHV1-8-IGHJ3   CARGQTYYDILTGPSDAFDIW
IGHV4-39-IGHJ5  CARSTGDWFDPW
IGHV3-9-IGHJ3   CANVPIYSSSYDAFDIW
IGHV3-23-IGHJ4  CAKDWELYYFDYW
IGHV3-23-IGHJ4  CAKDRGYTGFGVYFDYW
IGHV4-39-IGHJ4  CARHLGYNNSWYPFDYW
IGHV1-2-IGHJ4   CAREGYNWNDEGRFDYW
IGHV3-23-IGHJ3  CAKSSGWYLSDAFDIW
IGHV4-39-IGHJ4  CARYLGYNSNWYPFDYW
IGHV3-23-IGHJ6  CAKEGCSSGCPYYYYGMDVW
IGHV3-23-IGHJ3  CAKWGPDAFDIW
IGHV3-11-IGHJ   CATSGGSP
IGHV3-11-IGHJ4  CARDGDGYNDYW
IGHV1-2-IGHJ4   CARRIGYSSGSEDYW
IGHV1-2-IGHJ4   CARDIAVPGHGDYW
IGHV6-1-IGHJ4   CASGGAVPGYYFDYW

In the first column, there are some items that are different. In the second column every item is the unique. The first column items need to match, and then the second column items need to be sorted by a minimum mismatch of 2. Ideally this would be good to do with Levenshtein module since I can put the max, but I need two strings. Is there a way to use Levenshtein on every item in a single list?

What I need to do is open this file (I think sorting it by length first may help, however I'm not sure). After all the items are grouped by length, I need to sort these items into groups that differ by 1 character (the strings before the third "-" need to be identical, where the string after the "-" should differ only by 1 char.)

I think the problem I'm having is in regards to generating a proper for loop to iterate over the items length.

The code I have thus far:

import sys
import os
import Levenshtein

inp = sys.argv[1] # Input file containing single column of items

with open(inp, "r") as f1:
        vj = [line.strip() for line in f1]

lengths = []
for k in vj:
        i = len(k)
        lengths.append(i)

lengths_sort = sorted(lengths, reverse = True)

uniq_len = []
for i in lengths_sort:
       if i not in uniq_len:
                uniq_len.append(i)

print uniq_len #For QC purposes

def get_new_list(strings, counts, outlist=[]):
        for s in strings:
                if len(s) == counts[0]:
                        outlist.append(s)
        return outlist

new_vj = get_new_list(vj, uniq_len, outlist=[])
print new_vj
ham = Levenshtein.hamming(new_vj[0], new_vj[1])
print ham

So the output I was looking for is good, but not yet complete:

[46, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18]
46
['IGHV3-30/33rn-IGHJ4-CAKDPSLSSMITFGGVIVTRGYFDYW', 'IGHV3-30/33rn-IGHJ4-CARDPSLSSMITFGGVIVTRGYFDYW']
1

There are two items of length 46 (coincidentally the strings before the third "-" is the same; great) and they differ by only one character between the two strings.

My trouble is, 1. how can I iterate through the numbers in the uniq_len list as unput for matching length in the "strings" list (see function in code). 2. I want to create a new list for each differing length. 3. If there are multiple items in each new list, all items much differ by only 1 character.

Note: The "-"'s were creating using UNIX paste -d- command with 3 files containing 1 column each to create this file. Would it be easier to paste these files together with a \\t as the delimiter to create 3 columns?

So, open the file, strip the lines, and one can then match up the 1st column, 2nd column, and see if the third column differs by one or more characters?

All help is appreciated.

Answer 1

Update: Modified to handle a variable number of "id" sub-fields and print the results as a single string. Note several test cases were added to the end of the input to have some with a different number of leading fields making up the id (ie 2 instead of 3).

I also renamed the num_mismatches() function hamming_distance() because that's what it is.

Using the following input:

IGHV3-23-IGHJ4-CAKDRGYTGYGVYFDYW
IGHV4-39-IGHJ4-CARHDILTGYSYYFDYW
IGHV3-23-IGHJ3-CAKSGGWYLSDAFDIW
IGHV4-39-IGHJ4-CARTGFGELGFDYW
IGHV1-2-IGHJ2-CARDSDYDWYFDLW
IGHV1-8-IGHJ3-CARGQTYYDILTGPSDAFDIW
IGHV4-39-IGHJ5-CARSTGDWFDPW
IGHV3-9-IGHJ3-CANVPIYSSSYDAFDIW
IGHV3-23-IGHJ4-CAKDWELYYFDYW
IGHV3-23-IGHJ4-CAKDRGYTGFGVYFDYW
IGHV4-39-IGHJ4-CARHLGYNNSWYPFDYW
IGHV1-2-IGHJ4-CAREGYNWNDEGRFDYW
IGHV3-23-IGHJ3-CAKSSGWYLSDAFDIW
IGHV4-39-IGHJ4-CARYLGYNSNWYPFDYW
IGHV3-23-IGHJ6-CAKEGCSSGCPYYYYGMDVW
IGHV3-23-IGHJ3-CAKWGPDAFDIW
IGHV3-11-IGHJ-CATSGGSP
IGHV3-11-IGHJ4-CARDGDGYNDYW
IGHV1-2-IGHJ4-CARRIGYSSGSEDYW
IGHV1-2-IGHJ4-CARDIAVPGHGDYW
IGHV6-1-IGHJ4-CASGGAVPGYYFDYW
IGHV1-2-CAREGYNWNDEGRFDYW
IGHV4-39-CARSTGDWFDPW
IGHV1-2-CARDSDYDWYFDLW

and this script:

from collections import defaultdict
from itertools import izip, tee
import os
import sys

# http://en.wikipedia.org/wiki/Hamming_distance#Algorithm_example
def hamming_distance(s1, s2):
    """ Count number of mismatched characters in equal length strings. """
    if not isinstance(s1, basestring): raise ValueError('s1 is not a string')
    if not isinstance(s2, basestring): raise ValueError('s2 is not a string')
    if len(s1) != len(s2): raise ValueError('string lengths do not match')
    return sum(a != b for a, b in izip(s1, s2))

def pairwise(iterable):  # itertools recipe
    "s -> (s0,s1), (s1,s2), (s2, s3), ..."
    a, b = tee(iterable)
    next(b, None)
    return izip(a, b)

inp = sys.argv[1]  # Input file

unique = defaultdict(list)
with open(inp, 'rb') as file:
    for fields in (line.strip().split('-') for line in file):
        id = '-'.join(fields[:-1])  # recombine all but last field into an id
        unique[id].append(fields[-1])  # accumulate ending fields with same id

for id in sorted(unique):
    final_fields = unique[id]
    final_fields.sort(key=lambda field: len(field))  # sort by length
    print id + ':' + '-'.join(final_fields)
    if len(final_fields) > 1:  # at least one pair to compare for mismatches?
        for a, b in pairwise(final_fields):
            if len(a) == len(b) and hamming_distance(a, b) < 2:
                print '  {!r} and {!r} differ by < 2 characters'.format(a, b)

Output:

IGHV1-2:CARDSDYDWYFDLW-CAREGYNWNDEGRFDYW
IGHV1-2-IGHJ2:CARDSDYDWYFDLW
IGHV1-2-IGHJ4:CARDIAVPGHGDYW-CARRIGYSSGSEDYW-CAREGYNWNDEGRFDYW
IGHV1-8-IGHJ3:CARGQTYYDILTGPSDAFDIW
IGHV3-11-IGHJ:CATSGGSP
IGHV3-11-IGHJ4:CARDGDGYNDYW
IGHV3-23-IGHJ3:CAKWGPDAFDIW-CAKSGGWYLSDAFDIW-CAKSSGWYLSDAFDIW
  'CAKSGGWYLSDAFDIW' and 'CAKSSGWYLSDAFDIW' differ by < 2 characters
IGHV3-23-IGHJ4:CAKDWELYYFDYW-CAKDRGYTGYGVYFDYW-CAKDRGYTGFGVYFDYW
  'CAKDRGYTGYGVYFDYW' and 'CAKDRGYTGFGVYFDYW' differ by < 2 characters
IGHV3-23-IGHJ6:CAKEGCSSGCPYYYYGMDVW
IGHV3-9-IGHJ3:CANVPIYSSSYDAFDIW
IGHV4-39:CARSTGDWFDPW
IGHV4-39-IGHJ4:CARTGFGELGFDYW-CARHDILTGYSYYFDYW-CARHLGYNNSWYPFDYW-CARYLGYNSNWYPFDYW
IGHV4-39-IGHJ5:CARSTGDWFDPW
IGHV6-1-IGHJ4:CASGGAVPGYYFDYW

Hope this update is also helpful...

Answer 2

Let assume that you want to sort some lines by A) length of column, B) Levenshtein distance of that column vs the col above or below.

The issue immediately arrises that levenshtein distance is between 2 relative objects; ie, Levenshtein(a, b) not a mono property like len(a) . The value of the levenshtein distance changes as the lines in the list change in position relative to each other.

Python 2.X supports the older cmp versus key argument for sort. It is inefficient, since it must be reevaluated every pass through the sort. However, Levenshtein sort is relative to the item above and below the entry.

As an example, lets make a matrix of your example data:

txt='''\
IGHV3-23-IGHJ4  CAKDRGYTGYGVYFDYW
IGHV4-39-IGHJ4  CARHDILTGYSYYFDYW
IGHV3-23-IGHJ3  CAKSGGWYLSDAFDIW
IGHV4-39-IGHJ4  CARTGFGELGFDYW
IGHV1-2-IGHJ2   CARDSDYDWYFDLW
IGHV1-8-IGHJ3   CARGQTYYDILTGPSDAFDIW
IGHV4-39-IGHJ5  CARSTGDWFDPW
IGHV3-9-IGHJ3   CANVPIYSSSYDAFDIW
IGHV3-23-IGHJ4  CAKDWELYYFDYW
IGHV3-23-IGHJ4  CAKDRGYTGFGVYFDYW
IGHV4-39-IGHJ4  CARHLGYNNSWYPFDYW
IGHV1-2-IGHJ4   CAREGYNWNDEGRFDYW
IGHV3-23-IGHJ3  CAKSSGWYLSDAFDIW
IGHV4-39-IGHJ4  CARYLGYNSNWYPFDYW
IGHV3-23-IGHJ6  CAKEGCSSGCPYYYYGMDVW
IGHV3-23-IGHJ3  CAKWGPDAFDIW
IGHV3-11-IGHJ   CATSGGSP
IGHV3-11-IGHJ4  CARDGDGYNDYW
IGHV1-2-IGHJ4   CARRIGYSSGSEDYW
IGHV1-2-IGHJ4   CARDIAVPGHGDYW
IGHV6-1-IGHJ4   CASGGAVPGYYFDYW'''

data=[line.split() for line in txt.splitlines()] 
# [['IGHV3-23-IGHJ4', 'CAKDRGYTGYGVYFDYW'], ['IGHV4-39-IGHJ4', 'CARHDILTGYSYYFDYW'], ['IGHV3-23-IGHJ3', 'CAKSGGWYLSDAFDIW'], ['IGHV4-39-IGHJ4', 'CARTGFGELGFDYW'], ['IGHV1-2-IGHJ2', 'CARDSDYDWYFDLW'], ['IGHV1-8-IGHJ3', 'CARGQTYYDILTGPSDAFDIW'], ['IGHV4-39-IGHJ5', 'CARSTGDWFDPW'], ['IGHV3-9-IGHJ3', 'CANVPIYSSSYDAFDIW'], ['IGHV3-23-IGHJ4', 'CAKDWELYYFDYW'], ['IGHV3-23-IGHJ4', 'CAKDRGYTGFGVYFDYW'], ['IGHV4-39-IGHJ4', 'CARHLGYNNSWYPFDYW'], ['IGHV1-2-IGHJ4', 'CAREGYNWNDEGRFDYW'], ['IGHV3-23-IGHJ3', 'CAKSSGWYLSDAFDIW'], ['IGHV4-39-IGHJ4', 'CARYLGYNSNWYPFDYW'], ['IGHV3-23-IGHJ6', 'CAKEGCSSGCPYYYYGMDVW'], ['IGHV3-23-IGHJ3', 'CAKWGPDAFDIW'], ['IGHV3-11-IGHJ', 'CATSGGSP'], ['IGHV3-11-IGHJ4', 'CARDGDGYNDYW'], ['IGHV1-2-IGHJ4', 'CARRIGYSSGSEDYW'], ['IGHV1-2-IGHJ4', 'CARDIAVPGHGDYW'], ['IGHV6-1-IGHJ4', 'CASGGAVPGYYFDYW']]

Now sort those rows on length of row[1] :

data.sort(key=lambda l: len(l[1])) 

Now sort based on the levenshtein distance of the entries. Note that you need to use cmp since levenstein is based on two values:

data.sort(cmp=levenshtein)

# [['IGHV3-11-IGHJ', 'CATSGGSP'], ['IGHV4-39-IGHJ5', 'CARSTGDWFDPW'], ['IGHV3-23-IGHJ3', 'CAKWGPDAFDIW'], ['IGHV3-11-IGHJ4', 'CARDGDGYNDYW'], ['IGHV3-23-IGHJ4', 'CAKDWELYYFDYW'], ['IGHV4-39-IGHJ4', 'CARTGFGELGFDYW'], ['IGHV1-2-IGHJ2', 'CARDSDYDWYFDLW'], ['IGHV1-2-IGHJ4', 'CARDIAVPGHGDYW'], ['IGHV1-2-IGHJ4', 'CARRIGYSSGSEDYW'], ['IGHV6-1-IGHJ4', 'CASGGAVPGYYFDYW'], ['IGHV3-23-IGHJ3', 'CAKSGGWYLSDAFDIW'], ['IGHV3-23-IGHJ3', 'CAKSSGWYLSDAFDIW'], ['IGHV3-23-IGHJ4', 'CAKDRGYTGYGVYFDYW'], ['IGHV4-39-IGHJ4', 'CARHDILTGYSYYFDYW'], ['IGHV3-9-IGHJ3', 'CANVPIYSSSYDAFDIW'], ['IGHV3-23-IGHJ4', 'CAKDRGYTGFGVYFDYW'], ['IGHV4-39-IGHJ4', 'CARHLGYNNSWYPFDYW'], ['IGHV1-2-IGHJ4', 'CAREGYNWNDEGRFDYW'], ['IGHV4-39-IGHJ4', 'CARYLGYNSNWYPFDYW'], ['IGHV3-23-IGHJ6', 'CAKEGCSSGCPYYYYGMDVW'], ['IGHV1-8-IGHJ3', 'CARGQTYYDILTGPSDAFDIW']]

If you want to change the two element sort cmp to a key type (for Python 3 that does not support cmp), you can produce comparison objects:

def cmp_to_key(mycmp):
    'Convert a cmp= function into a key= function'
    class K(object):
        def __init__(self, obj, *args):
            self.obj = obj
        def __lt__(self, other):
            return mycmp(self.obj, other.obj) < 0
        def __gt__(self, other):
            return mycmp(self.obj, other.obj) > 0
        def __eq__(self, other):
            return mycmp(self.obj, other.obj) == 0
        def __le__(self, other):
            return mycmp(self.obj, other.obj) <= 0
        def __ge__(self, other):
            return mycmp(self.obj, other.obj) >= 0
        def __ne__(self, other):
            return mycmp(self.obj, other.obj) != 0
    return K 

The you call sort using data.sort(key=cmp_to_key(levenshtein))