While read line is very slow

Question

I want to create thumbs for large list of images. The problem is, it appears that while read line is very slow with large lists. one solution that i can think of is to create files that contain max 500 lines and then read them one by one. but is there any smart solution for this problem?

while read line; do
  if [ -e "$line" ] && [ ! -z "$line" ]; then
              ...
  fi
}
done <<< "$imagesList"

Answer 1

Your problem is that you're using a HERE string to read in these lines via <<< . This will be slow if the HERE string is very large.

If $imgageList is a file, you can do a file redirect and this will be a lot faster:

while read line
do
    if [ -e "$line" -a ! -z "$line" ]
    then
       ...
    fi
done < "$imagesList_file"  # Redirect from a file.

You might be able to do this:

echo "$imagesList" | while read line
    do
    if [ -e "$line" -a ! -z "$line" ]
    then
       ...
    fi
done

But, I would be worried about overloading the command line. In Linux/Unix systems, this is defined in the /usr/include/sys/syslimits.h or /usr/include/syslimits.h . It's 1024 * 256 on my system or 262,144 bytes. This sounds like a lot, but can be deceptive. File names can be quite long -- especially if you include the directory path in them. This is usually long enough to pass when you're testing, but fail when you really, really are depending upon it to work. And. it fails silently. You never know that the last few file names were dropped off.

Your best bet is to create a file with the list of image names instead of loading them up in an environment variable.

Answer 2

Let's try again:

echo $imagelist | xargs --delimiter=\  --max-args=1 -i echo "Do something with {}"

--- 8< First version --- 8< ---

Try something like this:

for IMG in $imageList; do
    echo "Do something with ${IMG}"
done

--- >8 First version --- >8 ---