Splitting and looping over live command output in Bash

Question

I am archiving and using split to produce several parts while also printing the output files (from split on STDERR, which I am redirecting to STDOUT). However the loop over the output data doesn't happen until after the command returns.

Is there anyway to actively split over the STDOUT output of a command before it returns?

The following is what I currently have, but it only prints the list of filenames after the command returns:

export IFS=$'\n'
for line in `data_producing_command | split -d -b $CHUNK_SIZE --verbose - $ARCHIVE_PREFIX 2>&1`; do
    FILENAME=`echo $line | awk '{ print $3 }'`
    echo "    - $FILENAME"
done

Answer 1

Try this:

data_producing_command | split -d -b $CHUNK_SIZE --verbose - $ARCHIVE_PREFIX 2>&1 | while read -r line
do
    FILENAME=`echo $line | awk '{ print $3 }'`
    echo "    - $FILENAME"
done

Note however that any variables set in the while loop will not preserve their values after the loop (the while loop runs in a subshell).

Answer 2

There's no reason for the for loop or the read or the echo. Just pipe the stream to awk:

... | split -d -b $CHUNK_SIZE --verbose - test 2>&1 |
 awk '{printf "    - %s\n", $3 }'

You are going to see some delay from buffering, but unless your system is very slow or you are very perceptive, you're not likely to notice it.

Answer 3

The command substitution needs ¹ to run before the for loop can start.

for item in $(command which produces items); do ...

whereas a while read -r can start consuming output as soon as the first line is produced (or, more realistically, as soon as the output buffer is full):

command which produces items |
while read -r item; do ...

¹ Well, it doesn't absolutely need to, from a design point of view, I suppose, but that's how it currently works.

As William Pursell already noted, there is no particular reason to run Awk inside a while read loop, because that's something Awk does quite well on its own, actually.

command which produces items |
awk '{ print "    - " $3 }'

Of course, with a reasonably recent GNU Coreutils split , you could simply do

split --filter='printf "   - %s\n" "$FILE"'; cat >"$FILE" ... options