The best way to filter a log by a dates range in python

Question

What\'s the best way to print log lines that match a datetime range. For example:

I would like to print only lines with dates from: 2012/09/30-00:00:10 to: 2012/09/

Answer 1

I am not sure of the performance implications (I suspect Tim's answer might be faster), but this approach works for any date range:

>>> def dates(start,end):
...     for i in range(int((end-start).seconds)):
...         yield start + datetime.timedelta(seconds=i)
...
>>> fmt = '%Y/%m/%d-%H:%M:%S'
>>> from_date = datetime.datetime.strptime('2012/09/30-00:00:10',fmt)
>>> till_date = datetime.datetime.strptime('2012/09/30-00:00:13',fmt)
>>> with open('file.log') as f:
...     for line in f:
...         if datetime.datetime.strptime(line.split()[0][:-4],fmt) in dates(fro
m_date,till_date):
...              print line
...
2012/09/30-00:00:10.526 log info
2012/09/30-00:00:10.995 log warning
2012/09/30-00:00:12.014 log warning

Answer 2

Assuming you're reading the log line by line:

import re
for line in log:
    if re.match("2012/09/30-00:00:1[0-3]", line):
        print line

Answer 3

Actullay, the log format allows to compare date strings without their conversion to datetime.

with open('mylog.log','r') as f:
    for line in f:
        d = line.split(" ",1)[0] 
        if d >= '2012/09/30-00:00:10' and d <= '2012/09/30-00:00:13':
            print line

Answer 4

.startswith() example:

prefixes = tuple("2012/09/30-00:00:1%d" % i for i in range(3))
with open('mylog.log', 'rb') as file:
    print ''.join(line for line in file if line.startswith(prefixes)),

You could optimize it by using a single static prefix and then testing preselected lines using a regex or datetime objects later.

If the lines are sorted by date in the input; you could break earlier without reading the whole file.

Answer 5

As per Tim's assumption you're reading a log file, line by line, then use itertools.

from itertools import dropwhile, takewhile

from_dt, to_td = '2012/09/30-00:00:10', '2012/09/30-00:00:13'
with open('logfile') as fin:
    of_interest = takewhile(lambda L: L <= to_td, dropwhile(lambda L: L < from_dt, fin))
    for line in of_interest:
        print line