How to use AWK with string as a RS?

Question

I want to use AWK, but I don't seem to get the first record right. I hope anyone can help to get it right.

I have this file, every record is 3 lines but sometimes it has 4 lines (so there is a $3 and $4). My goal is to print all three lines of each record, and if there is a forth line I want also to print the first 2 lines with the forth (without the 3rd).

My strategy is to use a string ("Sequence: ") as a RS, and new line ("\\n") for FS.

My file looks like this:

Sequence: X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

With the following code I get a messed up first record, because the string is in the beginning of the file as well.

awk '{ RS="Sequence: "; FS="\n" }
{
if ($4 != "" )
    print $1,"\n",$2,"\n",$3,"\n",$1,"\n",$2,"\n",$4
else
    print $1,"\n",$2,"\n",$3 ;
}' short.txt > test 

With output:

Sequence:
 X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 from:
 Sequence:
 X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 1
Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

So I thought I should remove the first "Sequence: " string from the input file, but that gives:

X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 from:
 1
 X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__
 from:
 to:
Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
 Start     End  Strand Pattern                 Mismatch Sequence
 184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
 M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
 Start     End  Strand Pattern                 Mismatch Sequence
 178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

So again the first record is messed up. Is there a solution to this problem? My expected output is as the last output (with or without the string "Sequence :"), but with the first record correct.

Answer 1

It sounds like this is what you're trying to do:

$ cat tst.awk
/^Sequence/ { if (NR>1) prt() }
{ rec[++cnt] = $0 }
END { prt() }
function prt() {
    print rec[1] ORS rec[2] ORS rec[3]
    if (cnt == 4) {
        print rec[1] ORS rec[2] ORS rec[4]
    }
    cnt=0
}

$ awk -f tst.awk file
Sequence: X92272_IGHV4-31*08_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: X92273_IGHV4-31*09_Homosapiens_F_V-REGION_140..429_290nt_1_____290+0=290_partialin3'__     from: 1   to: 290
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: Z14235_IGHV4-31*10_Homosapiens_F_V-REGION_140..438_299nt_1_____299+0=299___     from: 1   to: 299
Start     End  Strand Pattern                 Mismatch Sequence
184     192       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
Sequence: AB019439_IGHV4-34*01_Homosapiens_F_V-REGION_59657..59949_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc
Sequence: M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
150     158       + pattern:AA[CT]NNN[AT]CN        . aatcaatca
Sequence: M99684_IGHV4-34*02_Homosapiens_F_V-REGION_311..603_293nt_1_____293+0=293___     from: 1   to: 293
Start     End  Strand Pattern                 Mismatch Sequence
178     186       + pattern:AA[CT]NNN[AT]CN        . aacccgtcc

Trying to use a RS for this just makes your life harder and the resulting code non-portable (gawk-only)

Answer 2

Your code can be easily fixed as:

BEGIN{ RS="Sequence: "; FS="\n" }
(NR==1){next}
{
if ($4 != "" )
    print $1,"\n",$2,"\n",$3,"\n",$1,"\n",$2,"\n",$4
else
    print $1,"\n",$2,"\n",$3 ;
}

The first record would be empty, that is why it is skipped with next .

The reason you had problems with your first record is that you defined RS and FS after the first record was read (ie not in a BEGIN block which occurs before anything is done at all)

But what you really want, just to be sure, is RS="(^|\\n)Sequence: " This just to be sure that it starts at the beginning of the line or the file.