How to recursively traverse a directory tree and find only files?

Question

I am working on a scp call to download a folder present on a remote system. Downloaded folder has subfolders and within these subfolders there are a bunch of files which I want to pass as arguments to a python script like this:

scp -r researcher@192.168.150.4:SomeName/SomeNameElse/$folder_name/ $folder_name/
echo "File downloaded successfully"
echo "Running BD scanner"
for d in $folder_name/*; do
        if [[ -d $d ]]; then
                echo "It is a directory"
        elif [[ -f $d ]]; then
                echo "It is a file"
                echo "Running the scanner :"
                 python bd_scanner_new.py /home/nsadmin/Some/bash_script_run_files/$d
        else
                echo "$d is invalid file"
                exit 1
        fi
done

I have added the logic to find if there are any directories and excluding them. However, I don't traverse down those directories recursively.

Partial results below:

File downloaded succesfully
Running BD scanner
It is a directory
It is a directory
It is a directory
Exiting

I want to improve this code so that it traverses all directories and picks up all files. Please help me with any suggestions.

Answer 1

You can use shopt -s globstar in Bash 4.0+:

#!/bin/bash

shopt -s globstar nullglob
cd _your_base_dir
for file in **/*; do
  # will loop for all the regular files across the entire tree
  # files with white spaces or other special characters are gracefully handled
  python bd_scanner_new.py "$file"
done

Bash manual says this about globstar :

If set, the pattern '**' used in a filename expansion context will match all files and zero or more directories and subdirectories. If the pattern is followed by a '/', only directories and subdirectories match.

More globstar discussion here: https://unix.stackexchange.com/questions/117826/bash-globstar-matching

Answer 2

Why go through the trouble of using globbing for file matching but rather use find with is meant for this by using a process-substitution ( <() ) with a while-loop.

#!/bin/bash

while IFS= read -r -d '' file; do
    # single filename is in $file
    python bd_scanner_new.py "$file"
done < <(find "$folder_name" -type f -print0)

Here, find does a recursive search of all the files from the mentioned path to any level of sub-directories below. Filenames can contain blanks, tabs, spaces, newlines. To process filenames in a safe way, find with -print0 is used: filename is printed with all control characters & terminated with NUL which then is read command processes with the same de-limit character.

Note; On a side note, always double-quote variables in bash to avoid expansion by shell.