bash：等待进程替换子shell完成

Question

How can bash wait for the subshell used in process substitution to finish in the following construct? (This is of course simplified from the real for loop and subshell which I am using, but it illustrates the intent well.)

for i in {1..3}; do
    echo "$i"
done > >(xargs -n1 bash -c 'sleep 1; echo "Subshell: $0"')
echo "Finished"

Prints:

Finished
Subshell: 1
Subshell: 2
Subshell: 3

Instead of:

Subshell: 1
Subshell: 2
Subshell: 3
Finished

How can I make bash wait for those subshells to complete?

UPDATE

The reason for using process substitution is that I'm wanting to use file descriptors to control what is printed to the screen and what is sent to the process. Here is a fuller version of what I'm doing:

for myFile in file1 file2 file3; do
    echo "Downloading $myFile"     # Should print to terminal
    scp -q $user@$host:$myFile ./  # Might take a long time
    echo "$myFile" >&3             # Should go to process substitution
done 3> >(xargs -n1 bash -c 'sleep 1; echo "Processing: $0"')
echo "Finished"

Prints:

Downloading file1
Downloading file2
Downloading file3
Finished
Processing: file1
Processing: file2
Processing: file3

Processing each may take much longer than the transfer. The file transfers should be sequential since bandwidth is the limiting factor. I would like to start processing each file after it is received without waiting for all of them to transfer. The processing can be done in parallel, but only a with a limited number of instances (due to limited memory/CPU). So if the fifth file just finished transferring but only the second file has finished processing, the third and fourth files should complete processing before the fifth file is processed. Meanwhile the sixth file should start transferring.

Answer 1

you could have the subshell create a file that the main shell waits for.

tempfile=/tmp/finished.$$
for i in {1..3}; do
    echo "$i"
done > >(xargs -n1 bash -c 'sleep 1; echo "Subshell: $0"'; touch $tempfile)
while ! test -f $tempfile; do sleep 1; done
rm $tempfile
echo "Finished"

Answer 2

Bash 4.4 lets you collect the PID of a process substitution with $! , so you can actually use wait , just as you would for a background process:

case $BASH_VERSION in ''|[123].*|4.[0123])
  echo "ERROR: Bash 4.4 required" >&2; exit 1;;
esac

# open the process substitution
exec {ps_out_fd}> >(xargs -n1 bash -c 'sleep 1; echo "Subshell: $0"'); ps_out_pid=$!

for i in {1..3}; do
  echo "$i"
done >&$ps_out_fd

# close the process substitution
exec {ps_out_fd}>&-

# ...and wait for it to exit.
wait "$ps_out_pid"

---

Beyond that, consider flock -style locking -- though beware of races:

for i in {1..3}; do
  echo "$i"
done > >(flock -x my.lock xargs -n1 bash -c 'sleep 1; echo "Subshell: $0"')

# this is only safe if the "for" loop can't exit without the process substitution reading
# something (and thus signalling that it successfully started up)

flock -x my.lock echo "Lock grabbed; the subshell has finished"

---

That said, given your actual use case, what you want should presumably look more like:

download() {
  for arg; do
    scp -q $user@$host:$myFile ./ || (( retval |= $? ))
  done
  exit "$retval"
}
export -f download

printf '%s\0' file1 file2 file3 |
  xargs -0 -P2 -n1 bash -c 'download "$@"' _

Answer 3

You can use bash coproc to hold a read-able filedescriptor to be closed when all process' children die:

coproc read                  # previously: `coproc cat`, see comments
for i in {1..3}; do
    echo "$i"
done > >(xargs -n1 bash -c 'sleep 1; echo "Subshell: $0"')
exec {COPROC[1]}>&-          # close my writing side
read -u ${COPROC[0]}         # will wait until all potential writers (ie process children) end
echo "Finished"

Answer 4

If this is to be run on a system where there is an attacker you should not use a temp file name that can be guessed. So based on @Barmar's solution here is one that avoids that:

tempfile="`tempfile`"
for i in {1..3}; do
    echo "$i"
done > >(xargs -n1 bash -c 'sleep 1; echo "Subshell: $0"'; rm "$tempfile")
while test -f "$tempfile"; do sleep 1; done
echo "Finished"

Answer 5

I think you are making it more complicated than it needs to be. Something like this works because the internal bash executions are a subprocess of the main process, the wait causes the process to wait until everything is finished before printing.

for i in {1..3}
do
    bash -c "sleep 1; echo Subshell: $i"  &
done
wait
echo "Finished"

Unix and derivatives (Linux) have the ability to wait for child (sub) processes but not grandchild processes such as occurred in your original. Some would consider the polling solution where you go back and check for completion to be vulgar since it does not use this mechanism.

The solution where the xargs PID was captured was not vulgar, just too complicated.