在Unix上删除文件中所有重复项

Question

I would like to remove both occurrences of duplicates from a file based on a number of columns. Here is a toy example:

Would like to delete all records that do not have uniqueness through the first 4 columns. So applying the awk script to:

    BLUE,CAR,RED,HOUSE,40
    BLUE,CAR,BLACK,HOUSE,20
    BLUE,CAR,GREEN,HOUSE,10
    BLUE,TRUCK,RED,HOUSE,40
    BLUE,TRUCK,GREEN,HOUSE,40
    BLUE,TRUCK,RED,HOUSE,40

Should result in

    BLUE,CAR,RED,HOUSE,40
    BLUE,CAR,BLACK,HOUSE,20
    BLUE,CAR,GREEN,HOUSE,10
    BLUE,TRUCK,GREEN,HOUSE,40

I have tried:

awk -F"," -v OFS="," '{cnt[$1,$2,$3,$4]++} END {for (rec in cnt) if (cnt[rec] == 1) print rec}' ss.txt

Which successfully removes both dupes, but does not apply the correct delimiter or print the whole record, resulting in:

    BLUECARREDHOUSE
    BLUETRUCKGREENHOUSE
    BLUECARBLACKHOUSE
    BLUECARGREENHOUSE

I prefer an awk solution but any portable solution is welcomed.

Answer 1

Given that you want the whole record for the records that are unique in the first 4 columns, this would do the job:

awk -F',' '{cnt[$1,$2,$3,$4]++;line[$1,$2,$3,$4] = $0}
           END {for (rec in cnt) if (cnt[rec] == 1) print line[rec]}' \
    ss.txt

Save the lines as well as the counts; get back what you entered. This gets painful if you have gigabyte files; there are ways to only save the unique lines if you want. This only saves the first version of each line, and deletes an entry when it is known to be non-unique. (Untested - but I believe it should work. Modified per comment from Ed Morton .)

awk -F',' '{ if (cnt[$1,$2,$3,$4]++ == 0)
                 line[$1,$2,$3,$4] = $0
             else
                 delete line[$1,$2,$3,$4]
           }
           END {for (rec in line) print line[rec]}' \
    ss.txt

If you only wanted the 4 key columns, then this just saves the 4 columns in the comma-separated format that you'll print:

awk -F',' '{cnt[$1,$2,$3,$4]++;line[$1,$2,$3,$4] = $1 "," $2 "," $3 "," $4}
           END {for (rec in cnt) if (cnt[rec] == 1) print line[rec]}' \
    ss.txt