pipe then hyphen (stdin) as an alternative to for loop

Question

I wrote a few sed an awk commands to extract a set of IDs that are associated with file names. I would like to run a set of commands using these filenames from id.txt

cat id.txt

14235.gz
41231.gz
41234.gz

I usually write for loops as follows:

for i in $(cat id.txt);
do 
    command <options> $i
done

I thought I could also do cat id.txt | command <options> - cat id.txt | command <options> -

Is there a way to pipe the output of cat, awk, sed, etc, line by line into a command?

Answer 1

Use a while read loop see Don't read lines wit for

while IFS= read -r line_in_text_file; do
 echo "$line_in_text_file"
done < id.txt

Answer 2

Commands don't usually get their filename arguments on standard input. Using - as an argument means to read the file contents from standard input instead of a named file, it doesn't mean to get the filename from stdin.

You can use command substitution to use the contents of the file as all the filename arguments to the command:

command <options> $(cat id.txt)

or you can use xargs

xargs command <options> < id.txt

Answer 3

Is there a way to pipe the output of cat, awk, sed, etc, line by line into a command?

Compound commands can be placed in a pipe, the syntax is not very strict. The usual:

awk 'some awk script' |
while IFS= read -r line; do
    echo "$line"
done |
sed 'some sed script'

I avoid reading input line by line using a while read - it's very slow. It's way faster to use awk scripts and other commands.

Command groups can be used to:

awk 'some awk script' |
{     # or '(', but there is no need for a subshell
      echo "header1,header2"
      # remove first line
      IFS= read -r first_line
      # ignore last line
      sed '$d'
} |
sed 'some sed script'

Remember that pipe command are run in a subshell, so variable changes will not affect parent shell.

Bash has process substitution extension that let's you run a while loop inside parent shell:

var=1
while IFS= read -r line; do
     if [[ "$line" == 2 ]]; then
        var=2
      fi
done <(
    seq 10 |
    sed '$d'
)
echo "$var"  # will output 2

Answer 4

xargs can do this

cat id.txt | xargs command

From xargs help

$ xargs --help
Usage: xargs [OPTION]... COMMAND [INITIAL-ARGS]...
Run COMMAND with arguments INITIAL-ARGS and more arguments read from input.

Mandatory and optional arguments to long options are also
mandatory or optional for the corresponding short option.
  -0, --null                   items are separated by a null, not whitespace;
                                 disables quote and backslash processing and
                                 logical EOF processing
  -a, --arg-file=FILE          read arguments from FILE, not standard input
  -d, --delimiter=CHARACTER    items in input stream are separated by CHARACTER,
                                 not by whitespace; disables quote and backslash
...