How to add timestamp to STDERR redirection

Question

In bash/ksh can we add timestamp to STDERR redirection?

E.g. myscript.sh 2> error.log

I want to get a timestamp written on the log too.

Answer 1

If you want to redirect back to stdout I found you have to do this:

myscript.sh >> >( while read line; do echo "$(date): ${line}"; done )

Not sure why I need the > in front of the (, so <(, but thats what works.

Answer 2

The devscripts package in Debian/Ubuntu contains a script called annotate-output which does that (for both stdout and stderr).

$ annotate-output make
21:41:21 I: Started make
21:41:21 O: gcc -Wall program.c
21:43:18 E: program.c: Couldn't compile, and took me ages to find out
21:43:19 E: collect2: ld returned 1 exit status
21:43:19 E: make: *** [all] Error 1
21:43:19 I: Finished with exitcode 2

Answer 3

If you're talking about an up-to-date timestamp on each line, that's something you'd probably want to do in your actual script (but see below for a nifty solution if you have no power to change it). If you just want a marker date on its own line before your script starts writing, I'd use:

( date 1>&2 ; myscript.sh ) 2>error.log

What you need is a trick to pipe stderr through another program that can add timestamps to each line. You could do this with a C program but there's a far more devious way using just bash.

First, create a script which will add the timestamp to each line (called predate.sh):

#!/bin/bash
while read line ; do
    echo "$(date): ${line}"
done

For example:

( echo a ; sleep 5 ; echo b ; sleep 2 ; echo c ) | ./predate.sh

produces:

Fri Oct  2 12:31:39 WAST 2009: a
Fri Oct  2 12:31:44 WAST 2009: b
Fri Oct  2 12:31:46 WAST 2009: c

Then you need another trick that can swap stdout and stderr, this little monstrosity here:

( myscript.sh 3>&1 1>&2- 2>&3- )

Then it's simple to combine the two tricks by timestamping stdout and redirecting it to your file:

( myscript.sh 3>&1 1>&2- 2>&3- ) | ./predate.sh >error.log

The following transcript shows this in action:

pax> cat predate.sh
    #!/bin/bash
    while read line ; do
        echo "$(date): ${line}"
    done
pax> cat tstdate.sh
    #!/bin/bash
    echo a to stderr then wait five seconds 1>&2
    sleep 5
    echo b to stderr then wait two seconds 1>&2
    sleep 2
    echo c to stderr 1>&2
    echo d to stdout
pax> ( ( ./tstdate.sh ) 3>&1 1>&2- 2>&3- ) | ./predate.sh >error.log
    d to stdout
pax> cat error.log
    Fri Oct  2 12:49:40 WAST 2009: a to stderr then wait five seconds
    Fri Oct  2 12:49:45 WAST 2009: b to stderr then wait two seconds
    Fri Oct  2 12:49:47 WAST 2009: c to stderr

As already mentioned, predate.sh will prefix each line with a timestamp and the tstdate.sh is simply a test program to write to stdout and stderr with specific time gaps.

When you run the command, you actually get "d to stdout" written to stderr (but that's your TTY device or whatever else stdout may have been when you started). The timestamped stderr lines are written to your desired file.

Answer 4

#!/bin/bash

DEBUG=1

LOG=$HOME/script_${0##*/}_$(date +%Y.%m.%d-%H.%M.%S-%N).log
ERROR=$HOME/error.log

exec 2> $ERROR
exec 1> >(tee -ai $LOG)

if [ $DEBUG = 0 ] ; then
    exec 4> >(xargs -i echo -e "[ debug ] {}")
else
    exec 4> /dev/null
fi

# test
echo " debug sth " >&4
echo " log sth normal "
type -f this_is_error
echo " errot sth ..." >&2
echo " finish ..." >&2>&4
# close descriptor 4
exec 4>&-

Answer 5

I like those portable shell scripts but a little disturbed that they fork/exec date(1) for every line. Here is a quick perl one-liner to do the same more efficiently:

perl -p -MPOSIX -e 'BEGIN {$!=1} $_ = strftime("%T ", localtime) . $_'

To use it, just feed this command input through its stdin:

(echo hi; sleep 1; echo lo) | perl -p -MPOSIX -e 'BEGIN {$|=1} $_ = strftime("%T ", localtime) . $_'

Answer 6

Redirections are taken in order. Try this:

Given a script -

$: cat tst
echo a
sleep 2
echo 1 >&2
echo b
sleep 2
echo 2 >&2
echo c
sleep 2
echo 3 >&2
echo d

I get the following

$: ./tst 2>&1 1>stdout | sed 's/^/echo $(date +%Y%m%dT%H%M%S) /; e'
20180925T084008 1
20180925T084010 2
20180925T084012 3

And as much as I dislike awk, it does avoid the redundant subcalls to date.

$: ./tst 2>&1 1>stdout | awk "{ print strftime(\"%Y%m%dT%H%M%S \") \$0; fflush() }"; >stderr

$: cat stderr
20180925T084414 1
20180925T084416 2
20180925T084418 3

$: cat stdout
a
b
c
d