bash: How do I ensure termination of process substitution used with exec?

Question

If I run

$#/bin/bash
for i in `seq 5`; do
    exec 3> >(sed -e "s/^/$i: /"; echo "$i-")
    echo foo >&3
    echo bar >&3
    exec 3>&-
done

then the result is not synchronous; it could be something like:

1: foo
1: bar
2: foo
2: bar
1-
3: foo
3: bar
2-
3-
4: foo
5: foo
4: bar
5: bar
4-
5-

How do I ensure that the process substitution >(...) is completed before proceeding to the next iteration?

Inserting sleep 0.1 after exec 3>&- helped, but it's inelegant, inefficient, and not guaranteed to always work.

EDIT: The example may look silly, but it was for illustration only. What I'm doing is reading a stream of input in a loop, feeding each line to a process which occasionally changes during the loop. Easier explained in code:

# again, simplified for illustration
while IFS= read line; do
    case $line in
    @*)
        exec 3>&-
        filename=${line:1}
        echo "starting $filename"
        exec 3> >(sort >"$filename"; echo "finished $filename")
        ;;
    *)
        echo "$line" >&3
        ;;
    esac
done
exec 3>&-

Answer 1

The following works in bash 4, using coprocesses:

#!/bin/bash
fd_re='^[0-9]+$'
cleanup_and_wait() {
    if [[ ${COPROC[1]} =~ $fd_re ]] ; then
        eval "exec ${COPROC[1]}<&-"
        echo "waiting for $filename to finish" >&2
        wait $COPROC_PID
    fi
}

while IFS= read -r line; do
    case $line in
    @*)
        cleanup_and_wait
        filename=${line:1}
        echo "starting $filename" >&2
        coproc { sort >"$filename"; echo "Finished with $filename" >&2; }
        ;;
    *)
        printf '%s\n' "$line" >&${COPROC[1]}
        ;;
    esac
done
cleanup_and_wait

For prior versions of bash, a named pipe can be used instead:

cleanup_and_wait() {
    if [[ $child_pid ]] ; then
      exec 4<&-
      echo "waiting for $filename to finish" >&2
      wait $child_pid
    fi
}

# this is a bit racy; without a force option to mkfifo,
# however, the race is unavoidable
fifo_name=$(mktemp -u -t fifo.XXXXXX)
if ! mkfifo "$fifo_name" ; then
  echo "Someone else may have created our temporary FIFO before we did!" >&2
  echo "This can indicate an attempt to exploit a race condition as a" >&2
  echo "security vulnarability and should always be tested for." >&2
  exit 1
fi

# ensure that we clean up even on unexpected exits
trap 'rm -f "$fifo_name"' EXIT

while IFS= read -r line; do
    case $line in
    @*)
        cleanup_and_wait
        filename=${line:1}
        echo "starting $filename" >&2
        { sort >"$filename"; echo "finished with $filename" >&2; } <"$fifo_name" &
        child_pid=$!
        exec 4>"$fifo_name"
        ;;
    *)
        printf '%s\n' "$line" >&4
        ;;
    esac
done
cleanup_and_wait

A few notes:

• It's safer to use printf '%s\n' "$line" than echo "$line"; if a line contains only -e, for instance, some versions of echo will do nothing with it.
• Using an EXIT trap for cleanup ensures that an unexpected SIGTERM or other error won't leave the stale fifo sitting around.
• If your platform provides a way to create a FIFO with an unknown name in a single, atomic operation, use it; this would avoid the condition that requires us to always test whether the mkfifo is successful.

Answer 2

Easy, just pipe everything into cat.

#!/bin/bash
for i in `seq 5`; do
  {
  exec 3> >(sed -e "s/^/$i: /"; echo "$i-")
  echo foo >&3
  echo bar >&3
  exec 3<&-
  }|cat
done

Here's the output:

1: foo
1: bar
1-
2: foo
2: bar
2-
3: foo
3: bar
3-
4: foo
4: bar
4-
5: foo
5: bar
5-

Answer 3

mkfifo tmpfifo
for i in `seq 5`; do
  { sed -e "s/^/$i: /"; echo "$i-";} <tmpfifo &
  PID=$!
  exec 3> tmpfifo
  echo foo >&3
  echo bar >&3
  exec 3>&-
  wait $PID
done
rm tmpfifo

Answer 4

The "obvious" answer is to get rid of the process substitution.

for i in `seq 5`; do
    echo foo | sed -e "s/^/$i: /"; echo "$i-"
    echo bar | sed -e "s/^/$i: /"; echo "$i-"
done

So the question becomes, do you really need to structure your code using process substitution? The above is much simpler than trying to synchronize an asynchronous construct.

Answer 5

Another user asks the same question, and receives an exhaustive answer here.