比较文件的连续列并返回不匹配元素的数量

Question

I have a text file which looks like this:

# sampleID  HGDP00511  HGDP00511   HGDP00512   HGDP00512   HGDP00513  HGDP00513   

M rs4124251       0       0            A            G          0          A

M rs6650104       0       A            C            T          0          0

M rs12184279      0       0            G            A          T          0

I want to compare the consecutive columns and return the number of matching elements. I want to do this in Python. Earlier, I did it using Bash and AWK (shell scripting), but its very slow, as I have huge data to process. I believe Python would be a faster solution to this. But, I am very new to Python and I already have something like this:

for line in open("phased.txt"):
    columns = line.split("\t")

    for i in range(len(columns)-1):
        a = columns[i+3]
        b = columns[i+4]
        for j in range(len(a)):
            if a[j] != b[j]:
                print j

which is obviously not working. As I am very new to Python, I don't really know what changes to make to get this to work. (This is code is completely wrong and I guess I could use difflib, etc. But, I have never proficiently coded in Python before, so, skeptical to proceed)

I want to compare and return the number of non matching elements in each column(starting from the third) to every other column in the file. I have 828 columns in totality. Hence I would need 828*828 number of outputs. (You can think of an*n matrix where the (i,j)th element would be the number of non matching elements between them. My desired output in case of the above snippet would be:

3 4: 1

3 5: 3

3 6: 3

......

4 6: 3
..etc

Any help on this would be appreciated. Thanks.

Answer 1

I highly recommend you use pandas for this rather than writing your own code:

import numpy as np
import pandas as pd
df = pd.read_csv("phased.txt")
match_counts = {(i,j): 
                   np.sum(df[df.columns[i]] != df[df.columns[j]]) 
                           for i in range(3,len(df.columns))
                           for j in range(3,len(df.columns))}

match_counts
{(6, 4): 3,
 (4, 7): 2,
 (4, 4): 0,
 (4, 3): 3,
 (6, 6): 0,
 (4, 5): 3,
 (5, 4): 3,
 (3, 5): 3,
 (7, 7): 0,
 (7, 5): 3,
 (3, 7): 2,
 (6, 5): 3,
 (5, 5): 0,
 (7, 4): 2,
 (5, 3): 3,
 (6, 7): 2,
 (4, 6): 3,
 (7, 6): 2,
 (5, 7): 3,
 (6, 3): 2,
 (5, 6): 3,
 (3, 6): 2,
 (3, 3): 0,
 (7, 3): 2,
 (3, 4): 3}

Answer 2

A Pure native python library way of solving this - let us know how it compares with bash 828 x 828 should be a walk in the park.

Element Column counts:

I purposely wrote this with a step in flipping of the sequences, for simplicity and illustrative purposes - you can improve it with changed logic or usages of class objects, function decorators maybe etc...

Code Python 2.7:

shiftcol = 2  # shift columns as first two are to be ignored
with open('phased.txt') as f:
    data = [x.strip().split('\t')[shiftcol:] for x in f.readlines()][1:]

# Step 1: Flipping the data first
flip = []
for idx, rows in enumerate(data):
    for i in range(len(rows)):
        if len(flip) <= i:
            flip.append([])
        flip[i].append(rows[i])

# Step 2: counts store in temp dictionary
for idx, v in enumerate(flip):
    for e in v:
        tmp = {}
        for i, z in enumerate(flip):
            if i != idx and e != '0':
                # Dictionary to store results
                if i+1 not in tmp:  # note has_key will be deprecated
                    tmp[i+1] = {'match': 0, 'notma': 0}
                tmp[i+1]['match'] += z.count(e)
                tmp[i+1]['notma'] += len([x for x in z if x != e])

        # results compensate for column shift..
        for key, count in tmp.iteritems():
            print idx+shiftcol+1, key+shiftcol, ': ', count

sample output

>>> 3 4 :  {'match': 0, 'notma': 3}
>>> 3 5 :  {'match': 0, 'notma': 3}
>>> 3 6 :  {'match': 2, 'notma': 1}
>>> 3 7 :  {'match': 2, 'notma': 1}
>>> 3 3 :  {'match': 1, 'notma': 2}
>>> 3 4 :  {'match': 1, 'notma': 2}
>>> 3 5 :  {'match': 1, 'notma': 2}