如何查找最新的修改文件并使用SHELL代码删除它们

Question

I need some help with a shell code. Now I have this code:

find $dirname -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-

This code finds duplicated files (with same content) in a given directory. What I need to do is to update it - find out latest (by date) modified file (from duplicated files list), print that file name and also give opportunity to delete that file in terminal.

Answer 1

Doing this in pure bash is a tad awkward, it would be a lot easier to write this in perl or python.

Also, if you were looking to do this with a bash one-liner, it might be feasible, but I really don't know how.

Anyhoo, if you really want a pure bash solution below is an attempt at doing what you describe.

Please note that:

• I am not actually calling rm, just echoing it - don't want to destroy your files
• There's a "read -u 1" in there that I'm not entirely happy with.

Here's the code:

#!/bin/bash

buffer=''

function process {
    if test -n "$buffer"
    then
        nbFiles=$(printf "%s" "$buffer" | wc -l)
        echo "================================================================================="
        echo "The following $nbFiles files are byte identical and sorted from oldest to newest:"
        ls -lt -c -r $buffer
        lastFile=$(ls -lt -c -r $buffer | tail -1)
        echo

        while true
        do
            read -u 1 -p "Do you wish to delete the last file $lastFile (y/n/q)? " answer
            case $answer in
                [Yy]* ) echo rm $lastFile; break;;
                [Nn]* ) echo skipping; break;;
                [Qq]* ) exit;;
                * ) echo "please answer yes, no or quit";;
            esac
        done
        echo
    fi
}

find . -type f -exec md5sum '{}' ';' |
sort                                 |
uniq --all-repeated=separate -w 33   |
cut -c 35-                           |
while read -r line
do
    if test -z "$line"
    then
        process
        buffer=''
    else
        buffer=$(printf "%s\n%s" "$buffer" "$line")
    fi
done
process

echo "done"

Answer 2

Here's a "naive" solution implemented in bash (except for two external commands: md5sum , of course, and stat used only for user's comfort, it's not part of the algorithm). The thing implements a 100% Bash quicksort (that I'm kind of proud of):

#!/bin/bash

# Finds similar (based on md5sum) files (recursively) in given
# directory. If several files with same md5sum are found, sort
# them by modified (most recent first) and prompt user for deletion
# of the oldest

die() {
   printf >&2 '%s\n' "$@"
   exit 1
}

quicksort_files_by_mod_date() {
    if ((!$#)); then
        qs_ret=()
        return
    fi
    # the return array is qs_ret
    local first=$1
    shift
    local newers=()
    local olders=()
    qs_ret=()
    for i in "$@"; do
        if [[ $i -nt $first ]]; then
            newers+=( "$i" )
        else
            olders+=( "$i" )
        fi
    done
    quicksort_files_by_mod_date "${newers[@]}"
    newers=( "${qs_ret[@]}" )
    quicksort_files_by_mod_date "${olders[@]}"
    olders=( "${qs_ret[@]}" )
    qs_ret=( "${newers[@]}" "$first" "${olders[@]}" )
}

[[ -n $1 ]] || die "Must give an argument"
[[ -d $1 ]] || die "Argument must be a directory"

dirname=$1

shopt -s nullglob
shopt -s globstar

declare -A files
declare -A hashes

for file in "$dirname"/**; do
    [[ -f $file ]] || continue
    read md5sum _ < <(md5sum -- "$file")
    files[$file]=$md5sum
    ((hashes[$md5sum]+=1))
done

has_found=0
for hash in "${!hashes[@]}"; do
    ((hashes[$hash]>1)) || continue
    files_with_same_md5sum=()
    for file in "${!files[@]}"; do
        [[ ${files[$file]} = $hash ]] || continue
        files_with_same_md5sum+=( "$file" )
    done
    has_found=1
    echo "Found ${hashes[$hash]} files with md5sum=$hash, sorted by modified (most recent first):"
    # sort them by modified date (using quicksort :p)
    quicksort_files_by_mod_date "${files_with_same_md5sum[@]}"
    for file in "${qs_ret[@]}"; do
      printf "   %s %s\n" "$(stat --printf '%y' -- "$file")" "$file"
    done
    read -p "Do you want to remove the oldest? [yn] " answer
    if [[ ${answer,,} = y ]]; then
       echo rm -fv -- "${qs_ret[@]:1}"
    fi
done

if((!has_found)); then
    echo "Didn't find any similar files in directory \`$dirname'. Yay."
fi

I guess the script is self-explanatory (you can read it like a story). It uses the best practices I know of, and is 100% safe regarding any silly characters in file names (eg, spaces, newlines, file names starting with hyphens, file names ending with a newline, etc.).

It uses bash's globs, so it might be a bit slow if you have a bloated directory tree.

There are a few error checkings, but many are missing, so don't use as-is in production! (it's a trivial but rather tedious taks to add these).

The algorithm is as follows: scan each file in the given directory tree; for each file, will compute its md5sum and store in associative arrays:

• files with keys the file names and values the md5sums.
• hashes with keys the hashes and values the number of files the md5sum of which is the key.

After this is done, we'll scan through all the found md5sum, select only the ones that correspond to more than one file, then select all files with this md5sum, then quicksort them by modified date, and prompt the user.

A sweet effect when no dups are found: the script nicely informs the user about it.

I would not say it's the most efficient way of doing things (might be better in, eg, Perl), but it's really a lot of fun, surprisingly easy to read and follow, and you can potentially learn a lot by studying it!

It uses a few bashisms and features that only are in bash version ≥ 4

Hope this helps!

Remark. If on your system date has the -r switch, you can replace the stat command by:

date -r "$file"

Remark. I left the echo in front of rm . Remove it if you're happy with how the script behaves. Then you'll have a script that uses 3 external commands :) .