How to add date string to each line of a continuously written log file

Question

Having a long running program that continuously writes to a logfile - how is it possible, disregarding any buffering issues, to add a date string to each line written to that file using a linux script?

I would imagine something like this:

tail -f logfile | ADD_DATE_TO_EACH_LINE > logfile2

The input would be something like that:

abc
def
ghi
jkl

The output should be similar to that:

2011-06-16 18:30:59 abc
2011-06-16 18:31:00 def
2011-06-16 18:35:21 ghi
2011-06-16 18:40:15 jkl

Answer 1

With perl:

command 2>&1 | perl -pe 'print scalar(localtime()), " ";'

With gawk:

command 2>&1 | awk '{ print strftime(), $0; fflush() }'

Replace command with tail -f logfile for your specific example. Or, perhaps you could just redirect the original program's stdout/stderr to the above pipe.

Answer 2

Try

tail -f logfile | while read line; do echo `date` "$line" ; done

Answer 3

You can try this

cat /etc/motd | xargs -d"\n" -I {} date +"%Y-%m-%d %H:%M:%S {}"

Example output:

2013-02-26 15:13:57 
2013-02-26 15:13:57 The programs included with the Debian GNU/Linux system are free software;
2013-02-26 15:13:57 the exact distribution terms for each program are described in the
2013-02-26 15:13:57 individual files in /usr/share/doc/*/copyright.
2013-02-26 15:13:57 
2013-02-26 15:13:57 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
2013-02-26 15:13:57 permitted by applicable law.

Answer 4

A little lengthy, but here is what I came up with:

tail -f logfile | sed -u 's/%/%%/g' | xargs -I {} date +"%Y-%m-%d %H:%M:%S {}"

Answer 5

Can you configure the long running program to write it's output to the standard output and not to the logfile? In this case it would be easy to pipe the output to a script that first writes the current timestamp and then the entry.

If that is impossible, it may help to periodically (e.g. every second) read the logfile content, copy each line to another file (adding the current timestamp) and then deleting the logfile. This may, however, impose losing logfile entries that are written between reading and deleting the file :(

Answer 6

Or you can use python ...

cat /dev/urandom | python -c "from __future__ import print_function; import sys; import datetime; map(lambda x: print(datetime.datetime.now(), x), [line for line in sys.stdin.readlines()])"

Or use gnu screen

screen -a ls -lh -L -U -X command

First you need to enable logging and timestamp on your ~/.screenrc.

logfile /tmp/screen-%S-%n.log
logtstamp on