In a shell (bash) How can I execute more than one command in a pipeline?

Question

I'm trying pipe the output of an awk command through more than one command at once in a Bash shell, following my knowledge I'm coming up with this:

awk '$13 ~ /type/ {print $15}' filename.txt | (wc -l || sort -u)

I want the result of the awk command to be both counted AND sorted, how can I accomplish that? Even with && command it doesn't work, it does execute the first command and then exits. I guess it is my knowledge of bash which is failing.

Thanks in advance.

Answer 1

If you want to send output to two different commands in a single line, you'll need to do process substituion.

Try this:

awk '$13 ~ /type/ {print $15}' filename.txt | tee >(wc -l >&2) | sort -u

This outputs the line count on stderr and the sorted output on stdout. If you need the line count on stdout, you can do that leave off the >&2 , but then it will be passed to the sort call and (most likely) sorted to the top of the output.

EDIT: corrected description of what happens based on further testing.

Answer 2

in that case, do your counting in awk , why the need for pipes? don't make it more complicated

awk '$13 ~ /type/ {print $15;c++}END{print c} ' filename.txt | sort -u

Answer 3

If the size in the output is not too large to fit in memory and you don't need the wc and sort commands to work in parallel for performance reasons, here's a relatively simple solution:

output=$(awk '$13 ~ /type/ {print $15}' filename.txt; echo a)
printf "%s" "${output%a}" | sort -u
printf "%s" "${output%a}" | wc -l

That complication with the the extra a is in case the awk command might print some empty lines at the end of the input, which the $() construct would strip. You can easily choose which of sort or wc should appear first.

---

Here's a way that works with any POSIX shell (ash, bash, ksh, zsh, ...) but only on systems that have /dev/fd (which includes reasonably recent Linux, *BSD and Solaris). Like Walter's similar construction using the simpler method available in bash, ksh93 and zsh , the output of wc and the output of sort may be intermixed.

{
  awk '$13 ~ /type/ {print $15}' filename.txt |
  tee /dev/fd3 |
  wc -l
} 3>&1 1>&3 | sort -u

---

If you both need to deal with intermediate output that doesn't comfortably fit in memory and don't want to have the output of the two commands intermixed, I don't think there's an easy way in a POSIX shell, though it should be doable with ksh or zsh coprocesses.

Answer 4

I think the bigger question is: what are you expecting the output to be?

If you're trying to do two things, then do two things:

awk '$13 ~ /type/ {print $15}' filename.txt > tempfile
wc -l < tempfile
sort -u < tempfile
rm tempfile

Answer 5

You want to use named pipes created with mkfifo in combination with tee. An example is at http://www.softpanorama.org/Tools/tee.shtml