检查多个 tsv 文件并从 python 中的每个 tsv 中删除所有相同的行

Question

i have three tsv files.

file 1:

1   Alice   24      
10  Bill    23
4   Ellen   24
9   Mike    30

file 2:

6  Julie   76
2  Bob     42
7  Tom     54
5  Frank   30
1  Alice   24

file 3:

3  Dave    68
8  Jerry   34
1  Alice   24
5  Frank   30
2  Bob     42

OUTPUT: My desire output is to drop all the rows in which first and second column's values are the same from any of those tsv files and keep other rows as it is.

file 1:

10  Bill    23
9   Mike    30
4   Ellen   24

file 2:

6  Julie   76
7  Tom     54

file 3:

3  Dave    68
8  Jerry   34

And my tsv files are headless. I have tried following code so far.

with open('file2.tsv') as check_file:
    check_set = set([row.split('\t')[0].strip().upper() for row in check_file])

with open('file1.tsv', 'r') as in_file, open('file3.tsv', 'w') as out_file:
    for line in in_file:
        if line.split('\t')[0].strip().upper() in check_set:
            out_file.write(line)

But i didnot got my desired three output files with this code. Any help will be appreciated. Thanks in advance.

Answer 1

You first need to read all your TSV files and count each occurrence of the first two columns. Python's Counter() can be used for this (which is based on a dictionary).

Whilst reading each row in, save it in a data dictionary where the keys are the filenames and the contents are lists of the first two values along with the raw rows. A defaultdict() is used to avoid having to add an entry if it doesn't already exist before appending a new entry.

After reading everything in, counts can now be used to determine if any given row has been seen only once, other values can be skipped over.

from collections import Counter, defaultdict

counts = Counter()      # hold counts of each first two value pairs
data = defaultdict(list)  # hold all data from all files

for tsv in ['file1.tsv', 'file2.tsv', 'file3.tsv']:
    with open(tsv) as f_tsv:
        for row in f_tsv:
            split = list(map(str.strip, row.split('\t')))
            key = tuple(split[:2])  # first and second column values
            counts[key] += 1
            data[tsv].append((key, row))

for tsv, key_rows in data.items():
    with open('x' + tsv, 'w') as f_tsv:
        for key, row in key_rows:
            if counts[key] == 1:
                f_tsv.write(row)

I would recommend you add print() statements to better understand what each of the variables holds, eg print(counts) and print(data)

Note: take out the 'x' + when ready, this was added to write the output files to slightly different filenames to avoid overwriting the original files whilst testing.