Build file path & expand wildcard from variables

Question

input_dir="~/path/to/dir/"
file_ext=".fastq.gz"
input_path="${input_dir}*${file_ext}"

echo "$input_path"
# returns: ~/path/to/dir/*.fastq.gz

for f in "$input_path" ; do echo "$f" ; done

# returns: ~/path/to/dir/*.fastq.gz

I was expecting a list of all files with the extension .fastq.gz.

Answer 1

Unfortunately, you can't store a glob pattern without losing information about which elements are and aren't intended to be quoted. Thus, two separate variables need to be used for the prefix and the suffix at glob expansion time , with the glob itself expanded unquoted between them:

input_dir=~/path/to/dir/ # ~ must not be in quotes; use $HOME if you want to quote it
file_ext=".fastq.gz"
for f in "${input_dir}"/*"${file_ext}"; do
  echo "Found $f"
done

Note the pattern -- "$prefix"/*"$suffix" -- with both expansions within quotes; this prevents those expanded portions from themselves being subject to string-splitting or glob expansion.

---

If you want to store a directive controlling how files are identified in a shell variable, consider an array storing find arguments:

# this is an array variable, with one entry per argument
find_args=( ~/path/to/dir -maxdepth 1 -name '*.fastq.gz' )

while IFS= read -r -d '' filename; do
  printf 'Found %q\n' "$filename"
done < <(find "${find_args[@]}" -print0)

---

Another option is to perform the expansion early on, and store its result in an array variable, iterating over the contents of that variable in your for loop:

input_dir=~/path/to/dir
file_ext=.fastq.gz
input_files=( "$input_dir"/*"$file_ext" )

for f in "${input_files[@]}"; do
  printf 'Found %q\n' "$f"
done

---

By the way -- be aware that if your glob expression matches no files, the glob will by default evaluate to the expression itself. It may thus be wise to check for whether your $f actually exists , to handle this case, or to enable the nullglob shell option (which disables this behavior) at the top of your script.

Thus:

 shopt -s nullglob # this prevents a non-matching glob from evaluating to itself

...or...

for f in "$input_dir"/*"$file_ext"; do
  # lack of quotes acceptable here because [[ ]] suppresses string-splitting and globbing
  [[ -e $f || -L $f ]] || { printf '%s\n' "$f not found" >&2; continue; }
  printf 'Found %q\n' "$f"
done