linux - sed 循环遍历文件中的行列表

Question

I am having some trouble looping through a list of lines in File A and using each line to find a match in File B and then to print out multiple lines in File B.

This is how File A looks like

Nitab4.5_0000062g0520.1

Nitab4.5_0000436g0070.1

Nitab4.5_0000375g0110.1

This is how File B looks like

Nitab4.5_0000062g0520.1 Zinc finger, CCHC-type, Fibronectin-binding A, N-terminal, Domain of unknown function DUF814, Protein of unknown function DUF3441 MVKVRMNTADVAAEVKCLRRLIGMRCSNVYDLSPKTYVFKLMNSSGVTESGESEKVLLLM ESGVRLHTTDYLRDKSNTPSGFTLKLRKHIRTRRLEDVRQLGYDRIVLFQFGLGANAHYV ILELYAQGNILLTDSDFMVMTLLRSHRDDDKGLAIMSRHRYPVEICRVFKRTTTEKLQAA LMSSAETDKNEGVEDNEQGNDGSDALQQKQGNRKNIKATDSTKKMIDGVRAKSPTLKVVL GEALGYGPALSEHIILDAGLVPNAKIGKGFELEGEMLHSLIEAVKQFEDWLEDVILGEKV PEGYILMQQKALSKKDSSMCNNGASEKMYDEFCPLLLNQFKSRDFMKFEAFNAALDEFYS KIESQRSEQQQKAKESTAMQKLNKIRTDQENRVVTLKQEVEHCIKTAELIEYNLEDVDAA ILAVRVALANGMSWEDLARMVKEEKRSGNPVAGLIDKLHLERNCMTLLLSNNLDEMDDDE KTQPVDKVEVDLALSAHANARRWYEMKKRQESKQEKTVTAHEKAFKAAERKTRLQLSQEK TVAVISHMRKVHWFEKFNWFVSSENYLVISGRDAQQNEMIVKRYMSKGDLYVHAELHGAS STVIKNHKPEMPIPPLTLNQAGCFTVCQSQAWDSKIVTSAWWVYPNQVSKTAPTGEYLTV GSFMIRGKKNFLPPHPLIMGFGILFRLDESSLGFHLNERRVRGEEEGLNDAEQSDPSLAI PDSDSEEELSMETSVDKDITDVPNDRSSVAGTSYEVQSNSLLSISDDKVTNSHNSSVKVN SINNDGLSDSLGIMATSGTSQLEDLIDRALEIGSSTASTKNHGVPPLLGSAGQ QDNEEKK VTQREKPYITKAERRKLKKGSDSTEGAPARQEKQSEKNQKAQKQCDEDVNNSKSGGGKVI RGQKGKLKKIKEKYADQDEEERRIRMALLASAGKVEKVDQTIQSEKVDAEPDKGAKATTG PEDASKICYKCKKVGHLSRDCQENSDESLQSTANGGDGHSLTSAGNAANDRDRIVMEEED IHEIGEEEKEKLNDVDYLTGNPLPNDILLYAVPVCGPYNALQSYKYRVKLVPGTVKKGKA AKTAMNLFSHMPEATSREKELMKACTDPELVAAVKGNVKITSAGLTQLKQKQKKSKKSNK AES

Nitab4.5_0000375g0110.1 Tetratricopeptide-like helical, NSF attachment protein, Tetratricopeptide repeat, Malate dehydrogenase, active site, Tetratricopeptide repeat-containing domain MGDQIARGEEFEKKAEKKLSGWGLFGSKHDDAADLFDKAANCFKLAKSWDQAGAVYVKVA NCYLKLDSKHEAAGAYANAAHCYKKTNTREAISCLEQAVHMFLDIGRLNMSARYYKEIAE LYEQEQNLEQAIIYYEKAADLFQSEDVTTSANQCKQKIAQFSAELEKYQRAIEIFEEIAR HSVNNNLLKYGVRGHLLNAGICQLCKGDVVAINNALERYQELDPTFSGTRECKLLVDLAA AIDEEDVAKFTGSVKEYDSMTKLDALRTTLLLRVKEALKAKELEEDDLT

Nitab4.5_0000062g0530.1 DNA polymerase, palm domain, DNA-directed DNA polymerase, family B, conserved site, DNA-directed DNA polymerase, family B, multifunctional domain, DNA-directed DNA polymerase, family B MARVTGVPISFLLARGQSIKVLSQLLRKARQRNLVIPNVKQAGSEQGTYEGATVLEARAG FYEKPIATLDFASLYPSIMMAYNLCYCTLVTPEEFHKLNLCEVDVNKTPSGEMFVKSDLQ KGILPEILEELLAARKRAKADLKEAKDPLVKAVLDGRQLALKISANSVYGFTGATVGQLP CLEISSSVTSYGRQMIEKTKKLVEDKFTVLKGYEHNAEVIYGDTDSVMVQFGVPTVEEAM KLGREAADHISETFIKPLRLEFEKIYYPYLLISKKRYAGLLWTNPDKHDKMDAKGELLAT

Nitab4.5_0005502g0010.1 CDC6, C-terminal domain, P-loop containing nucleoside triphosphate hydrolase, Cell division protein Cdc6/18, Winged helix-turn-helix DNA-binding domain MPTIPVRRSPRISGGSKVAGQTVSRNEIGVSTPSKRKIRSDSTTEDNVVTSTLTPSPMEI SPCKWKSPRRCVNDSPKSPLNANRGDKTINLSKSPVKRRLSESFLEKPIWNPRDMEQLNA VKEALHVSRAPSNLVCRQVEQNRVLEFCKQAVKIEKAGSLYVCGCPGTGKSLSMEKVKEV LVNWADESGFQAPDILSVNCTSLSNTSDIFGKMLDKIQPRRKLNCSTAPLQYLQKMFSEK QQPAGTKMLLIVADELDYLITKDKVVLHELFMLTTSPFSRFILIGIANAIDLADRFLPKL QSMNCEYFPSCKPAVITFCAYSKDQIISILQQRFEKVASASGDMRKALWVCRLVNIAARL ADHSLTKSAIEMLEAEIRDSISSLDLPSLHGRVSYQHRDGACDKSPIHESNVVRVDHVAI ALSKAYRSPVVDTIQSLPQHQQIILCSAVKLFRGKKKDATIGELNISYLDVCKSTLIPPV GIMELSSMCRVLGDQGILKVGKAREEKLSRVTLKVDEADITFALQA

Nitab4.5_0005502g0020.1 MVIEEQCDDEGVQPYIEQLMDGQNYSQAQTHDGQSNDFNNSADTEIQQNDDSGKTIDVQI NSRNQFIGKEGRKLASFLGIVARTPELTPLQCKKWD

Nitab4.5_0005502g0030.1 MINERLRNNSERLNDHPPQSVAWEGDVYSQVLKNKKSGYVRGNIDLEDSSNEVKRLEQKV IELTKLNGKQNEEMSSMKPELLWMRKVMCKIAPNELYMSQNINEISIGQVTQIQKFKTFV LKH

Nitab4.5_0005502g0040.1 Ribosomal protein L10/acidic P0, Ribosomal protein L10/L12 MAVKVTKAEKKVNYDKKLCKLLDTYQQILIVGADNVGSNQLQMIRKGLRGDSIVLMGKNT MMKRSIRIHAEKTGNNAFLALIPCLVGNVGLIFTRGDLKEVSDEVSKYKVGAPARVGLVA PIDVVVPPGNTGLDPSQTSFFQVLNIPTKINKGTVEITIPVEIIKKGEKVGSSESALLSK LGIKPFSYGLIVQFVYDSGSVFSPEVLDLTEDDLIAKFAAGLSNVVGLSMLLSYPTLAAI PHMFINGYKNVLSFAIATEYSFPQAEKVKEYLKDPSKFATAIAAPVATKPAVKPATAKEE KKEEPAEEDDDDFVGGLFD

I wanted to print out the description lines (lines starting with >NitabXXXX) and the following amino acid sequences (the capitalized letters) in File B if the gene IDs (Nitab4.5_xxxxx) were found in File A.(In file B, the amino acid sequences were separated in multiple lines)

Here is the code I have come up with so far

while IFS= read -r Gene_ID; do sed -n '/$Gene_ID/,/>Nitab4.5/p' File B | sed '$d'; done < File A 

The code worked with a specified Gene ID and no loop. But I was not able to get it to work after adding the loop. I am new to Linux and sed. Hope someone could point out the mistake and help me correct the code. Thank you!!

Answer 1

Your question is a bit confusing, but could it be that this simple command is what you're looking for?

grep -f FILE_A -A 1 FILE_B

The options do the following:

-f FILE
Obtain patterns from FILE, one per line. The empty file contains zero patterns, and therefore matches nothing.
-A NUM
Print NUM lines of trailing context after matching lines. Places a line containing a group separator (described under --group-separator) between contiguous groups of matches.

Answer 2

First, let's try to print the third entry in FileB . (I am calling it FileB instead of File B because spaces in file names are big headaches.)

sed -n '/Nitab4.5_0000062g0530.1/,/>Nitab4.5/p' FileB
>Nitab4.5_0000062g0530.1 DNA polymerase, palm domain, DNA-directed DNA polymerase, family B, conserved site, DNA-directed DNA polymerase, family B, multifunctional domain, DNA-directed DNA polymerase, family B
MARVTGVPISFLLARGQSIKVLSQLLRKARQRNLVIPNVKQAGSEQGTYEGATVLEARAG
FYEKPIATLDFASLYPSIMMAYNLCYCTLVTPEEFHKLNLCEVDVNKTPSGEMFVKSDLQ
KGILPEILEELLAARKRAKADLKEAKDPLVKAVLDGRQLALKISANSVYGFTGATVGQLP
CLEISSSVTSYGRQMIEKTKKLVEDKFTVLKGYEHNAEVIYGDTDSVMVQFGVPTVEEAM
KLGREAADHISETFIKPLRLEFEKIYYPYLLISKKRYAGLLWTNPDKHDKMDAKGELLAT

>Nitab4.5_0005502g0010.1 CDC6, C-terminal domain, P-loop containing nucleoside triphosphate hydrolase, Cell division protein Cdc6/18, Winged helix-turn-helix DNA-binding domain

It picked up the first line of the next entry. So instead of terminating at ">Nitab4.5", let's terminate at the empty line:

sed -n '/Nitab4.5_0000062g0530.1/,/^$/p' FileB
>Nitab4.5_0000062g0530.1 DNA polymerase, palm domain, DNA-directed DNA polymerase, family B, conserved site, DNA-directed DNA polymerase, family B, multifunctional domain, DNA-directed DNA polymerase, family B
MARVTGVPISFLLARGQSIKVLSQLLRKARQRNLVIPNVKQAGSEQGTYEGATVLEARAG
FYEKPIATLDFASLYPSIMMAYNLCYCTLVTPEEFHKLNLCEVDVNKTPSGEMFVKSDLQ
KGILPEILEELLAARKRAKADLKEAKDPLVKAVLDGRQLALKISANSVYGFTGATVGQLP
CLEISSSVTSYGRQMIEKTKKLVEDKFTVLKGYEHNAEVIYGDTDSVMVQFGVPTVEEAM
KLGREAADHISETFIKPLRLEFEKIYYPYLLISKKRYAGLLWTNPDKHDKMDAKGELLAT

Now to do it with a variable:

line=Nitab4.5_0000062g0530.1; sed -n '/$line/,/^$/p' FileB

We get nothing, because the shell passed $line to sed, and sed has its own ideas about what that means. To get the shell to expand the variable before passing it to sed, we must use double quotes:

line=Nitab4.5_0000062g0530.1; sed -n "/$line/,/^$/p" FileB
>Nitab4.5_0000062g0530.1 DNA polymerase, palm domain, DNA-directed DNA polymerase, family B, conserved site, DNA-directed DNA polymerase, family B, multifunctional domain, DNA-directed DNA polymerase, family B
MARVTGVPISFLLARGQSIKVLSQLLRKARQRNLVIPNVKQAGSEQGTYEGATVLEARAG
FYEKPIATLDFASLYPSIMMAYNLCYCTLVTPEEFHKLNLCEVDVNKTPSGEMFVKSDLQ
KGILPEILEELLAARKRAKADLKEAKDPLVKAVLDGRQLALKISANSVYGFTGATVGQLP
CLEISSSVTSYGRQMIEKTKKLVEDKFTVLKGYEHNAEVIYGDTDSVMVQFGVPTVEEAM
KLGREAADHISETFIKPLRLEFEKIYYPYLLISKKRYAGLLWTNPDKHDKMDAKGELLAT

If this is satisfactory, we can start on the loop. Always start with something simple:

while read line; do echo $line; done < FileA
Nitab4.5_0000062g0520.1

Nitab4.5_0000436g0070.1

Nitab4.5_0000375g0110.1

Those empty lines are a pain, so let's remove them. We can do that several ways, but since we're using sed anyway, let's use sed:

sed '/^$/d' FileA | while read line; do echo $line; done
Nitab4.5_0000062g0520.1
Nitab4.5_0000436g0070.1
Nitab4.5_0000375g0110.1

Now we put it all together:

sed '/^$/d' FileA | while read line; do sed -n "/$line/,/^$/p" FileB; done 

Answer 3

Thank you for updating your input file. If awk is your option, would you please try the following:

awk '
    BEGIN {RS=ORS="\n\n"; FS="\n"}
    NR==FNR {
        for (i=1; i<=NF; i++) nitab[$i]
        next
    }
    {
        if (match($1, /^>[^[:blank:]]+/)) {
            str = substr($1, 0, RLENGTH)
            if (str in nitab) print
        }
    }
' FileA FileB

Output:

>Nitab4.5_0000062g0520.1 Zinc finger, CCHC-type, Fibronectin-binding A, N-terminal, Domain of unknown function DUF814, Protein of unknown function DUF3441
MVKVRMNTADVAAEVKCLRRLIGMRCSNVYDLSPKTYVFKLMNSSGVTESGESEKVLLLM
ESGVRLHTTDYLRDKSNTPSGFTLKLRKHIRTRRLEDVRQLGYDRIVLFQFGLGANAHYV
ILELYAQGNILLTDSDFMVMTLLRSHRDDDKGLAIMSRHRYPVEICRVFKRTTTEKLQAA
LMSSAETDKNEGVEDNEQGNDGSDALQQKQGNRKNIKATDSTKKMIDGVRAKSPTLKVVL
GEALGYGPALSEHIILDAGLVPNAKIGKGFELEGEMLHSLIEAVKQFEDWLEDVILGEKV
PEGYILMQQKALSKKDSSMCNNGASEKMYDEFCPLLLNQFKSRDFMKFEAFNAALDEFYS
KIESQRSEQQQKAKESTAMQKLNKIRTDQENRVVTLKQEVEHCIKTAELIEYNLEDVDAA
ILAVRVALANGMSWEDLARMVKEEKRSGNPVAGLIDKLHLERNCMTLLLSNNLDEMDDDE
KTQPVDKVEVDLALSAHANARRWYEMKKRQESKQEKTVTAHEKAFKAAERKTRLQLSQEK
TVAVISHMRKVHWFEKFNWFVSSENYLVISGRDAQQNEMIVKRYMSKGDLYVHAELHGAS
STVIKNHKPEMPIPPLTLNQAGCFTVCQSQAWDSKIVTSAWWVYPNQVSKTAPTGEYLTV
GSFMIRGKKNFLPPHPLIMGFGILFRLDESSLGFHLNERRVRGEEEGLNDAEQSDPSLAI
PDSDSEEELSMETSVDKDITDVPNDRSSVAGTSYEVQSNSLLSISDDKVTNSHNSSVKVN
SINNDGLSDSLGIMATSGTSQLEDLIDRALEIGSSTASTKNHGVPPLLGSAGQQDNEEKK
VTQREKPYITKAERRKLKKGSDSTEGAPARQEKQSEKNQKAQKQCDEDVNNSKSGGGKVI
RGQKGKLKKIKEKYADQDEEERRIRMALLASAGKVEKVDQTIQSEKVDAEPDKGAKATTG
PEDASKICYKCKKVGHLSRDCQENSDESLQSTANGGDGHSLTSAGNAANDRDRIVMEEED
IHEIGEEEKEKLNDVDYLTGNPLPNDILLYAVPVCGPYNALQSYKYRVKLVPGTVKKGKA
AKTAMNLFSHMPEATSREKELMKACTDPELVAAVKGNVKITSAGLTQLKQKQKKSKKSNK
AES

>Nitab4.5_0000375g0110.1 Tetratricopeptide-like helical, NSF attachment protein, Tetratricopeptide repeat, Malate dehydrogenase, active site, Tetratricopepti
de repeat-containing domain
MGDQIARGEEFEKKAEKKLSGWGLFGSKHDDAADLFDKAANCFKLAKSWDQAGAVYVKVA
NCYLKLDSKHEAAGAYANAAHCYKKTNTREAISCLEQAVHMFLDIGRLNMSARYYKEIAE
LYEQEQNLEQAIIYYEKAADLFQSEDVTTSANQCKQKIAQFSAELEKYQRAIEIFEEIAR
HSVNNNLLKYGVRGHLLNAGICQLCKGDVVAINNALERYQELDPTFSGTRECKLLVDLAA
AIDEEDVAKFTGSVKEYDSMTKLDALRTTLLLRVKEALKAKELEEDDLT

[Explanations]

• The BEGIN block assigns the input/output record reparators to double newlines and the field separator to a newline. It enables to handle the paragraph (a group of the description line and the amino acid lines) as a record.
• The condition FR==FNR returns TRUE while reading the 1st file in the argument list (= FileA) only. The idiom is useful to switch the procedure depending on the input files.
• The loop for (i=1; i<=NF; i++) nitab[$i] stores each line of FileA in an array nitab .
• The next statement match($1, /^>[^[:blank:]]+/) extracts the >NitabXXX substring of a recored in FileB which corresponds to the lines of FileA.
• Then the variable str is assigned to the substring.
• If str matches any entries of the array nitab , then print the record.