is there a way other than read/echo to read one line from a file descriptor to stdout without closing the fd?

Question

I ran into a situation where I was doing:

outputStuff |
filterStuff |
transformStuff |
doMoreStuff |
endStuff > endFile

I want to be able to insert some debug tracing stuff in a fashion like:

tee debugFile.$((debugNum++)) |

but obviously the pipes create subshells, so I wanted to do this instead.

exec 5< <(seq 1 100)
outputStuff |
tee debugFile.$(read -u 5;echo $REPLY;) |
filterStuff |
tee debugFile.$(read -u 5;echo $REPLY;) |
transformStuff |
tee debugFile.$(read -u 5;echo $REPLY;) |
doMoreStuff |
endStuff > endFile

ie, I want the debug line I insert to be identical, so I don't have to worry about stepping on various stuff. The read/REPLY echo seems really ugly.. I suppose I could wrap in a function.. but is there a way to read one line from a file descriptor to stdout without closing the fd (like a head -1 would close the fd is I did head -1 <&3)

Answer 1

Put all the reads inside a single compound statement, and redirect input from descriptor 5 for that .

{ outputStuff    | tee debugFile.$(read -u 5;echo $REPLY;) |
  filterStuff    | tee debugFile.$(read -u 5;echo $REPLY;) |
  transformStuff | tee debugFile.$(read -u 5;echo $REPLY;) |
  doMoreStuff    |
  endStuff
} 5< <(seq 1 100) > endFile

Now, file descriptor 5 is opened once (and closed once), and each call to read gets successive lines from that descriptor.

(You can also simplify this a bit; unless you are providing outputStuff with input via the keyboard, there doesn't seem to be a need to use file descriptor 5 instead of standard input, since only outputStuff is reading from standard input. All the other programs are reading their standard input via the pipeline.)

Answer 2

I tried several things, but in the end @Etan Reisner proved to me (unintentionally) that even if there is a way to do what you asked (clever, Etan), it's not what you actually want. If you want to be sure to read the numbers back sequentially then the reads have to be serialized, which the commands in a pipeline are not.

Indeed, that applies to your original approach as well, since command substitutions are performed in subshells. I think you could reliably do it like this, though:

debugum=1
eval "
    outputStuff |
    tee debugFile.$((debugNum++)) |
    filterStuff |
    transformStuff |
    doMoreStuff |
    tee debugFile.$((debugNum++)) |
    endStuff > endFile
"

That way all the substitutions are performed by the parent shell, on the string, before any of the commands is launched.

Answer 3

Here's an approach that doesn't need seq at all:

Define a function that constructs your pipeline recursively.

buildWithDebugging() {
  local -a nextCmd=( )
  local debugNum=$1; shift
  while (( $# )) && [[ $1 != '|' ]]; do
    nextCmd+=( "$1" )
    shift
  done
  if (( ${#nextCmd[@]} )); then
    "${nextCmd[@]}" \
      | tee "debugFile.$debugNum" \
      | buildWithDebugging "$((debugNum + 1))" "$@"
  else
    cat # noop
  fi
}

...and, to use it:

buildWithDebugging 0 \
  outputStuff '|'
  filterStuff '|'
  transformStuff '|'
  doMoreStuff '|'
  endStuff > endFile

A more secure version would use pipeline components done in the style of Pascal strings rather than C strings -- which is to say, instead of using literal | s, preceding each string of commands with its length:

buildWithDebugging 0 \
  1 outputStuff
  3 filterStuff filterArg filterSecondArg
  2 transformStuff filterArg
  1 doMoreStuff
  endStuff > endFile

Building this should be a completely trivial exercise for the reader. :)

Answer 4

At the cost of evenly padding your numbers you can do this with dd though you don't end up with a nicer looking command for your trouble. =)

exec 5< <(seq -w 10 -1 1)
echo -n |
{ echo "d 1:$(dd bs=1 count=3 2>/dev/null <&5)"; cat; } |
{ echo "d 2:$(dd bs=1 count=3 2>/dev/null <&5)"; cat; } |
{ echo "d 3:$(dd bs=1 count=3 2>/dev/null <&5)"; cat; } |
{ echo "d 4:$(dd bs=1 count=3 2>/dev/null <&5)"; cat; } |
cat

You can also just use a shorter variable with read : read -u 5 a;echo $a but that only saves you two characters.