根据bash中文件名中的子字符串复制最新的更新文件

Question

I have to archive some files (based on date which is there in file) from a folder but there can be multiple files with same name (substring). I have to copy only the latest one to a saperate folder.

for eg.
20180730.abc.xyz2.jkl.20180729.164918.csv.gz

In this -> 20180730 and 20180729 are representing date from which I have to search by (first date) 20180730. This part is done.

The searching part which i wrote is :

for FILE in $SOURCE_DIR/$BUSINESS_DT*
{
do
 # Here I have to search if this FILENAME exists and if yes, then copy that  latest file
 cp "${FILE}" $TARGET_DIR/
done

Now I have to search if the same SOURCE_DIR contains a file with the name similar to 20180730.abc.xyz2.jkl. and if it exists then I have to copy it. so basically, I have to extract the portion abc.xyz2.jkl. I can't use cut with fields as the filename could either be like abc.xyz2.jkl or abc.xyz. The portion is variable and can also have numberthe last two numbers are also variable and can change. Some eg are:

20180730.abc.xyz2.jkl.20170729.890789.csv.gz
 20180730.abc.xyz2.20180729.121212.csv.gz
 20180730.ab.xy.20180729.11111.csv.gz

Can anybody please help me in doing that. I tried find and cut but didn't got required results.

Many Thanks

Answer 1

Python might be a better choice for implementing something like this, but here is a bash example. You can use sed positional parameter to extract the portion of the filename that you want. Then use an associative array to store the filename of the newest file containing the substring found. Once that's done, you can go back and do the copy operations. Here is an example which extracts the string between the two 8-digit numbers and periods. This sed expression may not work for your complete data set, but it works for the 3 examples you gave. Also this won't handle cases where one unique identifier is a subset of another unique identifier.

declare -A LATEST
for FILE in $SOURCE_DIR/$BUSINESS_DT*
do
     # Extract the substring unique identifier
    HASH=$(echo "${FILE}" |  sed "s/[0-9]\{8\}\.\(.*\)\.[0-9]\{8\}.*$/\\1/g")

    # If this is the first time on this unique identifier,
    # then get the latest matching file
    if [ ${LATEST[${HASH}]}abc == abc ]
    then
        LATEST[${HASH}]=$(find . -type f -name '*${HASH}*' -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" ")
    fi
done

for FILE in "${!LATEST[@]}"
do
    cp "${FILE}" $TARGET_DIR/
done