当行名称与排序不匹配时，基于UNIX中的一列内部连接两个文件

Question

I'm running into an issue trying to join two datasets in unix and could use your help. I've spent a long time searching the forum for a solution but turned up empty handed.

I have a list of accession numbers in one dataset and need to convert these to gene symbols. In order to do so I downloaded the gene2accession.gz from NCBI . The uncompressed file is ~7Gb so first I cut out the accession and gene symbol from this dataset

cut -f 2,16 gene2accession > accession2genesymbol

There are ~70 million lines as per wc -l accession2genesymbol with many duplicates so I removed these with sort accession2genesymbol | uniq sort accession2genesymbol | uniq which resulted in ~20 million lines.

Now normally I would do an inner_join() using the dplyr package in R (return all rows from x where there are matching values in y, and all columns from x and y); however, this dataset is far too large for me to work with.

Here is a sample of the unsorted accession2genesymbol dataset:

100000492       mafaa
1000004 XCC3444
110047139       LOC110047139
110047140       LOC110047140
9951915         LOAG_14435
9951916         LOAG_14436
999999          gndA
999             CDH1
9               NAT1

A short example of the unsorted the Accessions looks like this (for the whole dataset -1,576 lines see the gist ):

Accessions
10047140
100913206
10092617
10190704
10190704
103471987
103471997
103472005
103472005
105990514
45006951
45006986
45006986
45007007
45007007
4501883
4501887
94721250
94721261
9558733
9845516
98986457
98986457
98986464
99028871
9910242
9951915
9966805
9966827
9966867
9994185

EDIT: Only accession 9951915 and 110047140 here have matches so my expected output would be:

9951915         LOAG_14435
110047140       LOC110047140

Not having worked with unix much for data manipulation/joining I searched Stack Overflow for similar problems.

For example this one . It's my understanding that unix join function can only be used if the files are sorted so I tried the following:

join -t "\t" <(dos2unix <accession) <(dos2unix <accession2genesymbol.txt)

Perhaps this is not working because I would need exactly the same row numbers in both datasets (ie if row 100 of dataset doesn't match row 100 of dataset2 it wont work) but perhaps I'm wrong and this didn't work for some other reason?

Perhaps awk is a better solution, so I tried a suggestion from this post :

awk '{a[$1]=a[$1] FS $2} END {for (i in a) print i a[i]}' accession accession2genesymbol | sort > file3

This produces a file with ~20 million lines and since my accession is only 9000 lines I would expect 9000 (or potentially fewer if those accessions no longer exist, for example).

I tried another awk solution from the first post:

awk -F, 'FNR==NR{a[$1];next}($1 in a){print $2}' accession accession2genesymbol > file3
awk: warning: escape sequence `\+' treated as plain `+'

But I end up with an empty file.

I'd appreciate an awk(ward) solution, python, or whatever would help me solve this problem. Thank you very much.

Answer 1

join should work for your case. since your input files don't have matches here is a made up example and using your map file

$ head file
100000009
100000061
100000030

$ join <(sed 1d map) <(sort file)
100000009 sema5bb+
100000030 btr24+
100000061 si:ch211-133n4.9+

assuming your map file is already sorted, you need to remove the header sed 1d and need to sort your input file . Note that sorting should be both numerical or lexical.

Another alternative, which doesn't require sorting is with grep

$ grep -wFf file map
100000009       sema5bb+
100000030       btr24+
100000061       si:ch211-133n4.9+

if the numbers and codes are not in the same format there won't be false matches.

Answer 2

We haven't seen a sample of your original gene2accession file yet but let's assume it's a tab-separated field with accession in the 2nd column and gene in the 16th (since that's what your cut is selecting) with a header line. Let's also assume that your Accessions file isn't absolutely enormous.

Given that, this script should do what you want:

awk -F'\t' 'NR==FNR{a[$1];next} ($2 in a) && !seen[$2]++{print $2, $16}' Accessions gene2accession

but you could try this to see if it's faster:

awk -F'\t' 'NR==FNR{a[$1];next} $2 in a{print $2, $16}' Accessions <(sort -u -t'\t' -k2,2 gene2accession)

and if it is and you want an intermediate file for the output of the sort to use in subsequent runs:

sort -u -t'\t' -k2,2 gene2accession > unq_gene2accession &&
awk -F'\t' 'NR==FNR{a[$1];next} $2 in a{print $2, $16}' Accessions unq_gene2accession