Compare two files using bash script and print detailed diff report

Question

I have 2 large files on Unix system which have thousands of rows and about 80 columns each. I have sorted the files based on group of unique keys so that we compare the same rows always. To ease of understanding I am giving only 3 rows and 7 columns here.

File 1:

d_report_ref_date="2021-03-31" system_id="VTX" contract_id="1130" credit_line_cd="ABC123" contract_id="ABC123" src_system_id="PRA" entity_cd="U0525"     
d_report_ref_date="2021-03-31" system_id="VTX" contract_id="1130" credit_line_cd="ABC124" contract_id="ABC124" src_system_id="PRA" entity_cd="U0526"     
d_report_ref_date="2021-03-31" system_id="VTX" contract_id="1130" credit_line_cd="ABC125" contract_id="ABC125" src_system_id="PRA" entity_cd="U0527" 

File2:

d_report_ref_date="2021-03-31" system_id="VTX" contract_id="1130" credit_line_cd="ABC123" contract_id="ABC123" src_system_id="PRA" entity_cd="U0525"     
d_report_ref_date="2021-03-31" system_id="VTX" contract_id="1130" credit_line_cd="ABC124" contract_id="ABC124" src_system_id="PRB" entity_cd="V0528"    
d_report_ref_date="2021-03-31" system_id="VTX" contract_id="1130" credit_line_cd="ABC125" contract_id="ABC125" src_system_id="PRA" entity_cd="U0530" 

Expected Output:

Mismatch in row 2 : file1.src_system_id=PRA file2.src_system_id=PRB, file1.entity_cd=U0526 file2.entity_cd=V0528 

Mismatch in row 3 : file1.entity_cd=U0527 file2.entity_cd=U0530

Is it possible to achieve this using bash scripting? I tried AWK which isn't giving me the desired output-

paste -d' ' file1 file2| 
  awk -F' ' '{w=NF/2; 
              for(i=1;i<=w;i++) 
                 if($i!=$(i+w)) printf "%d %d %s %s", NR,i,$i,$(i+w); 
              print ""}'

Thanks in Advance !!!

Answer 1

Using any awk in any shell on every Unix box:

$ cat tst.awk
BEGIN { FS="[= ]" }
NR==FNR {
    for (i=1; i<NF; i+=2) {
        file1[NR,i] = $(i+1)
    }
    next
}
{
    msg = sep = ""
    for (i=1; i<NF; i+=2) {
        if ( $(i+1) != file1[FNR,i] ) {
            msg = msg sep " " ARGV[1] "." $i "=" file1[FNR,i] " " FILENAME "." $i "=" $(i+1)
            sep = ","
        }
    }
    if ( msg != "" ) {
        print "Mismatch in row " FNR " :" msg ORS
    }
}

$ awk -f tst.awk file1 file2
Mismatch in row 2 : file1.src_system_id="PRA" file2.src_system_id="PRB", file1.entity_cd="U0526" file2.entity_cd="V0528"

Mismatch in row 3 : file1.entity_cd="U0527" file2.entity_cd="U0530"

The above assumes:

1. Your quoted strings cannot contain = or blanks
2. Every tag present in a row of file1 is also present in the same row of file2
3. The tags are always present in the same order in a given row
4. You can have multiple duplicate tags in a given row

Answer 2

Take a look at wdiff , something like this might work:

$ wdiff -w$'\e[31m' -x $'\e[0m' -y $'\e[32m' -z $'\e[0m' file1 file2

The options -wxyz is to define prefix and suffix for deletion and insertions respectively. In this case we do a naive attempt to color deletions red, and insertions green.

Answer 3

A bit late to the party, but you can do some kind of "nested-diff" where the first diff captures the different rows and put every column from them in a line of its own. Then, you do another diff to capture the exact different columns. It is pure bash, and uses only bash loops, grep , sed and diff .

$ for f in $(diff file1.txt file2.txt | grep -e "<"); do echo $f; done > left && for f in $(diff file1.txt file2.txt | grep -e ">"); do echo $f; done > right && diff -y left right | grep "|" | sed "s/\t>/-----/g"
<                                 |-----
src_system_id="PRA"                       | src_system_id="PRB"
entity_cd="U0526"                         | entity_cd="V0528"
<                                 |-----
entity_cd="U0527"                         | entity_cd="U0530"