如何加快检查文件是否存在于 bash 中

Question

I'm new at Bashing and wrote a code to check my photos files but find it very slow and gets a few empty returns checking 17000+ photos. Is there any way to use all 4 cpus running this script and so speed it up

Please help

#!/bin/bash
readarray -t array < ~/Scripts/ourphotos.txt
totalfiles="${#array[@]}"
echo $totalfiles
i=0
ii=0
check1=""
while : 
do

check=${array[$i]}
if [[ ! -r $( echo $check ) ]] ; then
    if [ $check = $check1 ]; then
     echo "empty "$check
    else
    unset array[$i]
    ii=$((ii + 1 ))
    fi
fi
if [ $totalfiles = $i ]; then
break
fi
i=$(( i + 1 ))
done 

if [ $ii -gt "1" ]; then
 notify-send -u critical $ii" files have been deleted or are unreadable"
 fi

Answer 1

It's a filesystem operation so multiple cores will hardly help. Simplification might:

while read file; do 
   i=$((i+1)); [ -e "$file" ] || ii=$(ii+1)); 
done < "$HOME/Scripts/ourphotos.txt"
#...

Two points:

• you don't need to keep the whole file in memory (no arrays needed)
• $( echo $check ) forks a proces. You generally want to avoid forking and execing in loops.

Answer 2

This is an old question, but a common problem lacking an evidence-based solution.

awk '{print "[ -e "$1" ] && echo "$2}' | parallel    # 400 files/s
awk '{print "[ -e "$1" ] && echo "$2}' | bash        # 6000 files/s
while read file; do [ -e $file ] && echo $file; done # 12000 files/s
xargs find                                           # 200000 files/s
parallel --xargs find                                # 250000 files/s
xargs -P2 find                                       # 400000 files/s
xargs -P96 find                                      # 800000 files/s

I tried this on a few different systems and the results were not consistent, but xargs -P (parallel execution) was consistently the fastest. I was surprised that xargs -P was faster than GNU parallel (not reported above, but sometimes much faster), and I was surprised that parallel execution helped so much — I thought that file I/O would be the limiting factor and parallel execution wouldn't matter much.

Also noteworthy is that xargs find is about 20x faster than the accepted solution, and much more concise. For example, here is a rewrite of OP's script:

#!/bin/bash

total=$(wc -l ~/Scripts/ourphotos.txt | awk '{print $1}')

# tr '\n' '\0' | xargs -0 handles spaces and other funny characters in filenames
found=$(cat ~//Scripts/ourphotos.txt | tr '\n' '\0' | xargs -0 -P4 find | wc -l)

if [ $total -ne $found ]; then
  ii=$(expr $total - $found)
  notify-send -u critical $ii" files have been deleted or are unreadable"
fi