git stderr output can't pipe

Question

I'm writing a graphical URI handler for git:// links with bash and zenity, and I'm using a zenity 'text-info' dialog to show git's clone output while it's running, using FIFO piping. The script is about 90 lines long, so I won't bother posting it here, but here's the most important lines:

git clone "$1" "$target" 2>&1 | cat >> /tmp/githandler-fifo &
cat /tmp/githandler-fifo | zenity --text-info --text='Cloning git repository' &

I'm using FIFO instead of a direct pipe to allow them to run asynchronously and allow for killing git if the zenity window is closed.

Problem is, the only line that appears from git's output is the first:

Initialized empty Git repository in /home/delan/a/.git/

The other lines with counting objects, etc. don't show or are shown on the terminal.

Current reason

The current consensus as to why this isn't working seems to be that cat is non-blocking and quits after the first line, only passing that to zenity and not the rest. My aim is to force blocking for the read, and have zenity's text info dialog show all output progressively.

git outputs progress messages (anything other than the "Initialized" message) on stderr, but the moment I try to pipe stderr to a file or to merge with stdout, the messages disappear.

Fix attempt 1

I've tried to write two blocking versions of cat's functions in C, bread and bwrite, like this:

#include <stdio.h>
main(int argc, char **argv) {
    int c;
    for (;;) {
        freopen(argv[1], "r", stdin);
        while ((c = getchar()) != EOF)
            putchar(c);
    }
}

#include <stdio.h>
main(int argc, char **argv) {
    int c;
    for (;;) {
        freopen(argv[1], "w", stdout);
        while ((c = getchar()) != EOF)
            putchar(c), fputs("writing", stderr);
    }
}

They work nicely because they block and don't quit on EOF, but it hasn't quite solved the problem yet. At the moment, using one, the other, or both, works in theory, but in practice, zenity shows nothing at all now.

Fix attempt 2

@mvds suggested that using a regular file, in combination with tail -f rather than cat, may do this. Surprised at such a simple solution (thanks!) I tried it but unfortunately, only the first line showed up in zenity and nothing else.

Fix attempt 3

After doing some strace'ing and inspecting git's source code, I realise that git outputs all its progress information (anything past the "Initialized" message) on stderr, and the fact that this is the first line and my assumption that it's because of cat quitting early on EOF was a coincidence/misguided assumption (git doesn't EOF until the program ends).

The situation seemed to become a lot simpler, as I shouldn't have to change anything from the original code (at the start of the question) and it should work. Mysteriously, however, the stderr output 'vanishes' when redirected - and this is only something that happens in git.

Test case? Try this, and see if you see anything in the file (you won't):

git clone git://anongit.freedesktop.org/xorg/proto/dri2proto 2> hurr

This goes against everything I know about stderr and redirection; I've even written a little C program that outputs on stderr and stdout to prove to myself that redirection just doesn't work for git.

Fix attempt 4

In line with Jakub Narębski's answer, as well as replies to emails I sent to the git mailing list, --progress is the option I need. Note that this option only works after the command, and not before clone.

Success!

Thank you very much for all your help. This is the fixed line:

git clone "$1" "$target" --progress > /tmp/githandler-fifo 2>&1 &

Answer 1

I think that at least some of progress reports gets silenced when output is not a terminal (tty). I'm not sure if it applies to your case, but try to pass --progress option to 'git clone' (i.e. use git clone --progress <repository>).

Though I don't know if it is what you wanted to have.

Answer 2

For one thing, the output redirection is parsed right-to-left, so

git clone "$1" "$target" 2>&1 > /tmp/githandler-fifo &

is not equal to

git clone "$1" "$target" > /tmp/githandler-fifo 2>&1 &

The latter will redirect stderr to stdout, and then stdout (including stderr) to the file. The former will redirect stdout to the file, and then show stderr on stdout.

As for piping to zenity (which I don't know), I think you may be making things overly complicated with the named pipe. Using strace may shed some light on the inner workings of the processes you're firing up. For the inexperienced, named pipes make things worse compared to normal pipes.

Answer 3

Given the experiment with the FIFO called 'a', I think the problem lies in the way zenity processes its input. What happens if you type into zenity from the keyboard? (Suspicion: it behaves as you'd want, reading to EOF.) However, it could be that zenity handles terminal input (tty input) using regular blocking I/O but uses non-blocking I/O for all other device types. Non-blocking I/O is fine for input from files; it is less desirable for input from pipes or FIFOs, etc. If it did use non-blocking I/O, zenity would get the first line of output, and then exit the loop thinking it was done because its second read attempt would indicate that there was nothing else immediately available.

Demonstrating that this is what is happening (or not) will be tricky. I would be looking to 'truss' or 'strace' or other system call monitor to track what zenity is doing.

As to workarounds...if the hypothesis is correct, then you'll need to persuade zenity that it is reading from a terminal and not a FIFO, so you'll probably need to rig up a pseudo-tty (or pty); the first process would write to the master end of the pty and you'd arrange for zenity to read from the slave end of the pty. You might still use the FIFO too - though it makes a long chain of command.