Bash: Head & Tail behavior with bash script

Question

Suppose I have following script:-

test.sh

#!/bin/bash
command1  #prints 5 lines
command2  #prints 3 lines

I run th

Answer 1

Nice finding. According to my tests it's exactly like you said. For example I have this script that just eats cpu, to let us spot it in top:

for i in `seq 10`
  do echo $i
  x=`seq 10000000`
done

Piping the script with head -n1 we see the command returning after the first line. This is the head behavior: it completed its work, so it can stop and return the control to you.

The input script should continue running but look what happens: when the head returns, its pid doesn't exist anymore. So when linux tries to send the output of the script to the head process, it does not find the process, so the script crashes and stops.

Let's try it with a python script:

for i in xrange(10):
    print i
    range(10000000)

When running it and piping to head you have this:

$ python -u test.py | head -n1
0
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    print i
IOError: [Errno 32] Broken pipe

The -u option tells python to automatically flush the stdin and stdout, as bash would do. So you see that the program actually stops with an error.

Answer 2

This is a fairly interesting issue! Thanks for posting it!

I assumed that this happens as head exits after processing the first few lines, so SIGPIPE signal is sent to the bash running the script when it tries to echo $x next time. I used RedX's script to prove this theory:

#!/usr/bin/bash
rm x.log
for((x=0;x<5;++x)); do
    echo $x
    echo $x>>x.log
done

This works, as You described! Using t.sh|head -n 2 it writes only 2 lines to the screen and to x.log. But trapping SIGPIPE this behavior changes...

#!/usr/bin/bash
trap "echo SIGPIPE>&2" PIPE
rm x.log
for((x=0;x<5;++x)); do
    echo $x
    echo $x>>x.log
done

Output:

$ ./t.sh |head -n 2
0
1
./t.sh: line 5: echo: write error: Broken pipe
SIGPIPE
./t.sh: line 5: echo: write error: Broken pipe
SIGPIPE
./t.sh: line 5: echo: write error: Broken pipe
SIGPIPE

The write error occurs as stdout is already closed as the other end of the pipe is closed. And any attempt to write to the closed pipe causes a SIGPIPE signal, which terminates the program by default (see man 7 signal). The x.log now contains 5 lines.

This also explains why /bin/echo solved the problem. See the following script:

rm x.log
for((x=0;x<5;++x)); do
    /bin/echo $x
    echo "Ret: $?">&2
    echo $x>>x.log
done

Output:

$ ./t.sh |head -n 2
0
Ret: 0
1
Ret: 0
Ret: 141
Ret: 141
Ret: 141

Decimal 141 = hex 8D. Hex 80 means a signal was received, hex 0D is for SIGPIPE. So when /bin/echo tried to write to stdout it got a SIGPIPE and it was terminated (as default behavior) instead of the bash running the script.

Answer 3

This is more of a comment then an answer but it is too big for a comment.

I tried following script:

#!/usr/bin/env bash

rm -f "test_head.log"
echo "1 line"
echo "1 line" >> "test_head.log"
echo "2 line"
echo "2 line" >> "test_head.log"
echo "3 line"
echo "3 line" >> "test_head.log"
echo "4 line"
echo "4 line" >> "test_head.log"
echo "5 line"
echo "5 line" >> "test_head.log"
echo "6 line"
echo "6 line" >> "test_head.log"
echo "7 line"
echo "7 line" >> "test_head.log"
echo "8 line"   
echo "8 line" >> "test_head.log"

Then i ran the script with:

./test_head.sh | head -n1

The cat output is (to my surprise):

1 line

I have no idea what is going on.

After reading @ymonad comment i tried it out and replace echo with /bin/echo and that solved the problem. I hope he can explain more about this behaviour.