Bash script to move specific line or lines in a file to other line with awk or sed?

Question

I see this being answered here: Moving matching lines in a text file using sed and here: How to move specified line in file to other place with regexp match (bash script)?

But I am having problems implementing it for my case.

I have a .csv file that contains:

Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2

Results:

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,GraphTests,37,26,11,70.00%

Mean Pass Rate for All Tests,,52,41,11,78.85%

I need to move the:

Mean Pass Rate for All Tests,,52,41,11,78.85%

up to right below "Results". So it would look like this:

Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2

Results:
Mean Pass Rate for All Tests,,52,41,11,78.85%

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,GraphTests,37,26,11,70.00%

The line numbers are not guaranteed. The "Files Used" section may include more files. Also, there are possibly other "Mean Pass Rate" lines depending on the tests being ran. For example:

Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2
    CLIENT FILE - /path/to/other/file/client.txt

Results:

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,ClientVarTests,12,12,0,100.00%
,GraphTests,37,26,11,70.00%

Mean Pass Rate for Client Tests,,27,27,0,100.00%
Mean Pass Rate for All Tests,,64,53,11,82.81%

In my script I tried awk:

MEAN_PASS_ALL="Mean Pass Rate for All Tests"

awk '/${MEAN_PASS_ALL}/{t=$0;next}1;/RESULTS/{print t}' ${CSV_FILE}

However all that did was print out the .csv file contents and did not move the line:

$ ./parse_results.sh
Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2

Results:

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,GraphTests,37,26,11,70.00%

Mean Pass Rate for All Tests,,52,41,11,78.85%

I also tried sed, but that gave errors:

sed "/.*Mean/d;/.*RESULTS/i"$(sed -n '/.*Mean/p' ${CSV_FILE})  ${CSV_FILE}

$ ./parse_results.sh
sed: can't read Pass: No such file or directory
sed: can't read Rate: No such file or directory
sed: can't read for: No such file or directory
sed: can't read All: No such file or directory
sed: can't read Tests,,52,41,11,78.85%: No such file or directory
Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2

Results:

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,GraphTests,37,26,11,70.00%

It looks like it removed the line, but it did not print it to where I want it, nor did it "save" the change.

If there are more than one line that contain "Mean Pass Rate", I would like them all moved to right below the "Results:"

What am I doing wrong here? How do I move the lines in the file and have the file saved with the changes?

Thanks!

Answer 1

You may use 2 pass:

awk 'FNR == NR {
   if ($0 ~ /^Mean Pass Rate /)
      mpr = mpr ORS $0
      next
}
/^Results:/ {
   $0 = $0 mpr
}
!/^Mean Pass Rate /' file file

Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2
    CLIENT FILE - /path/to/other/file/client.txt

Results:
Mean Pass Rate for Client Tests,,27,27,0,100.00%
Mean Pass Rate for All Tests,,64,53,11,82.81%

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,ClientVarTests,12,12,0,100.00%
,GraphTests,37,26,11,70.00%

Answer 2

Could you please try following, written and tested in GNU awk . In case you want to perform inplace editing of Input_file then append > temp && mv temp Input_file to following solution.

awk '
FNR==NR{
  if($0~/Mean Pass Rate/){
    val=(val?val ORS:"")$0
  }
  next
}
/Results:/{
  print $0 ORS val
  val=""
  next
}
/Mean Pass Rate/{
  next
}
1
' Input_file  Input_file

Explanation: Adding detailed explanation for above.

awk '                              ##Starting awk program from here.
FNR==NR{                           ##Checking condition FNR==NR which will be TRUE when 1st Input_file is being read.
  if($0~/Mean Pass Rate/){         ##Checking condition if line has Mean Pass Rate then do following.
    val=(val?val ORS:"")$0         ##Creating val variable which is keep on having current line in it.
  }
  next                             ##next will skip all further statements from here.
}
/Results:/{                        ##Checking condition if line has Results: then do following.
  print $0 ORS val                 ##Printing current line ORS and val here.
  val=""                           ##Nullifying val here.
  next                             ##next will skip all further statements from here.
}
/Mean Pass Rate/{                  ##Checking condition if line has Mean Pass Rate then do following.
  next                             ##next will skip all further statements from here.
}
1                                  ##1 will print current line here.
' Input_file  Input_file           ##Mentioning Input_file names here.

Answer 3

Use ed to edit files, not sed . It's meant to work with them, doing things like moving lines around to arbitrary spots in the file:

ed -s "$CSV_FILE" <<'EOF'
/^Results:$/ka
g/^Mean Pass Rate/m'a\
ka
w
EOF

This first mar k s the Results: line, and then for each line starting with Mean Pass Rate , m oves that line to after the mark and changes the mark point to that most recently moved line (Otherwise they'd end up in reverse order), and then w rites the modified file back out to disk.

Running it on your sample file with two Mean Pass Rate lines results in it being edited to:

Unit Tests
Model: A
Tue Sep  8 22:52:24 MDT 2020
User: br
Files Used:
    DATA FILE - /path/to/file/data.txt
    TEST TARBALL - /path/to/tarball/my_test.tar.bz2
    CLIENT FILE - /path/to/other/file/client.txt

Results:
Mean Pass Rate for Client Tests,,27,27,0,100.00%
Mean Pass Rate for All Tests,,64,53,11,82.81%

Module,Test Suite,Total Tests,Tests Passed,Tests Failed,Pass Rate

Individual Test Run,ClientTests,15,15,0,100.00%
,ClientVarTests,12,12,0,100.00%
,GraphTests,37,26,11,70.00%

Answer 4

tac myFile | awk '
/^Mean Pass Rate/ { move=(!move)?$0:move ORS $0;next }
/^Results:/ { $0=move ORS $0 }
1' - | tac