How can I tail a log file in Python?

Question

I\'d like to make the output of tail -F or something similar available to me in Python without blocking or locking. I\'ve found some really old code to do that here, but I\'

Answer 1

Ideally, I'd have something like tail.getNewData() that I could call every time I wanted more data

We've already got one and itsa very nice. Just call f.read() whenever you want more data. It will start reading where the previous read left off and it will read through the end of the data stream:

f = open('somefile.log')
p = 0
while True:
    f.seek(p)
    latest_data = f.read()
    p = f.tell()
    if latest_data:
        print latest_data
        print str(p).center(10).center(80, '=')

For reading line-by-line, use f.readline(). Sometimes, the file being read will end with a partially read line. Handle that case with f.tell() finding the current file position and using f.seek() for moving the file pointer back to the beginning of the incomplete line. See this ActiveState recipe for working code.

Answer 2

Another option is the tailhead library that provides both Python versions of of tail and head utilities and API that can be used in your own module.

Originally based on the tailer module, its main advantage is the ability to follow files by path i.e. it can handle situation when file is recreated. Besides, it has some bug fixes for various edge cases.

Answer 3

You can also use 'AWK' command.
See more at: http://www.unix.com/shell-programming-scripting/41734-how-print-specific-lines-awk.html
awk can be used to tail last line, last few lines or any line in a file.
This can be called from python.

Answer 4

So, this is coming quite late, but I ran into the same problem again, and there's a much better solution now. Just use pygtail:

Pygtail reads log file lines that have not been read. It will even handle log files that have been rotated. Based on logcheck's logtail2 (http://logcheck.org)

Answer 5

You could use the 'tailer' library: https://pypi.python.org/pypi/tailer/

It has an option to get the last few lines:

# Get the last 3 lines of the file
tailer.tail(open('test.txt'), 3)
# ['Line 9', 'Line 10', 'Line 11']

And it can also follow a file:

# Follow the file as it grows
for line in tailer.follow(open('test.txt')):
    print line

If one wants tail-like behaviour, that one seems to be a good option.

Answer 6

Adapting Ijaz Ahmad Khan's answer to only yield lines when they are completely written (lines end with a newline char) gives a pythonic solution with no external dependencies:

def follow(file) -> Iterator[str]:
    """ Yield each line from a file as they are written. """
    line = ''
    while True:
        tmp = file.readline()
        if tmp is not None:
            line += tmp
            if line.endswith("\n"):
                yield line
                line = ''
        else:
            time.sleep(0.1)


if __name__ == '__main__':
    for line in follow(open("test.txt", 'r')):
        print(line, end='')