awk +过滤日志文件

Question

I used the following nice awk command in order to filter duplicate lines

example:

cat LogFile | awk '!seen[$0]++'

the problem is that in some cases we need to filter duplicate lines in spite some fields are different and they no so important

for example

LogFile:

 [INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08
 [INFO],[02/Jun/2014-19:31:25],EXE,ds1a,INHT VERION , 1.4.4.3-08
 [INFO],[02/Jun/2014-19:32:40],EXE,ds1a,INHT VERION , 1.4.4.3-08

please take a look on this file - LogFile

I need to remove the duplicate lines from the third delimiter " , " until the end of the line ,

and no matter what is before the third delimiter

so finally I should get this filtered file: ( should get always the first one in the list )

    [INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08

so please help me to complete my task

how to filter the LofFile from the third delimiter " , " , and ignore the fields: [INFO],[...........],EXE,

Remark – implantation can be also with perl one liner line

Answer 1

With GNU awk for gensub():

$ awk '!seen[gensub(/([^,]*,){3}/,"","")]++' file
[INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08

With any awk that supports RE intervals (most modern awks):

$ awk '{key=$0; sub(/([^,]*,){3}/,"",key)} !seen[key]++' file
[INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08

Answer 2

Using a perl one-liner:

perl -lne '$k = s/(.*?,){3}//r; print if !$seen{$k}++' file.log

Outputs:

[INFO],[02/Jun/2014-19:30:45],EXE,ds1a,INHT VERION , 1.4.4.3-08

Explanation:

Switches :

• -l : Enable line ending processing. ( Only needed if last line of log file is missing the new line )
• -n : Creates a while(<>){..} loop for each line in your input file.
• -e : Tells perl to execute the code on command line.

Code :

• $k = s/(.*?,){3}//r : Save everything after the third comma in the variable $k
• print if !$seen{$k}++ : Print the line if the key is not seen before.

Answer 3

使用autosplit的方式略有不同：

perl -aF, -ne'print unless $seen{"@F[3..$#F]"}++' logfile.txt

Answer 4

你可以有：

awk 'BEGIN{FS=OFS=","}{o=$0;$1=$2=$3=""}!seen[$0]++{print o;}' ...