在目录中按名称查找文件的重复项-Linux

Question

I have a folder which contains sub folders and some more files in them.

The files are named in the following way

abc.DEF.xxxxxx.dat

I'm trying to find the duplicate files only matching 'xxxxxx' in the above pattern ignoring the rest. The extension .dat doesn't change. But the length of abc and DEF might change. The order of separation by periods also doesn't change.

I'm guessing I need to use Find in the following way

find -regextype posix-extended -regex '\w+\.\w+\.\w+\.dat'

I need help coming up with the regular expression. Thanks.

Example: For a file named 'epg.ktt.crwqdd.dat', I need to find duplicate files containing 'crwqdd'.

Answer 1

You can use awk for that:

find /path -type f -name '*.dat' | awk -F. 'a[$4]++'

Explanation:

Let find give the following output:

./abd.DdF.TTDFDF.dat
./cdd.DxdsdF.xxxxxx.dat
./abc.DEF.xxxxxx.dat
./abd.DdF.xxxxxx.dat
./abd.DEF.xxxxxx.dat

Basically, spoken with the words of a computer, you want to count the occurrences of a pattern between .dat and the next dot and print those lines where pattern appeared at least the second time.

To achieve this we split the file names by the . what gives us 5(!) fields:

 echo ./abd.DEF.xxxxxx.dat | awk -F. '{print $1 " " $2 " " $3 " " $4  " " $5}'
  /abd DEF xxxxxx dat

Note the first, empty field. The pattern of interest is $4 .

To count the occurrences of a pattern in $4 we use an associative array a and increment it's value on each occurrence. Unoptimized, the awk command will look like:

... | awk -F. '{{if(a[$4]++ > 1){print}}'

However, you can write an awk program in the form:

CONDITION { ACTION }

What will give us:

... | awk -F. 'a[$4]++ > 1 {print}'

print is the default action in awk . It prints the whole current line. As it is the default action it can be omitted. Also the >1 check can be omitted because awk treats integer values greater than zero as true . This gives us the final command:

... | awk -F. 'a[$4]++' 

To generalize the command we can say the pattern of interest isn't the 4th column, it is the next to last column. This can be expressed using number of fields in awk its NF :

... | awk -F. 'a[$(NF-1)]++'

Output:

./abc.DEF.xxxxxx.dat
./abd.DdF.xxxxxx.dat
./abd.DEF.xxxxxx.dat