Need to do a daily log rotation (0utc) using Python

Question

I'm an admitted noob to Python. I've written a little logger that takes data from the serial port and writes it to a log file. I've got a small procedure that opens the file for append, writes, then closes. I suspect this might not be the best way to do it, but it's what I've figured out so far.

I'd like to be able to have it automagically perform a log-rotate at 00 UTC, but so far, my attempts to do this with RotatingFileHandler have failed.

Here's what the code looks like:

import time, serial, logging, logging.handlers,os,sys
from datetime import *

CT12 = serial.Serial()
CT12.port = "/dev/ct12k"
CT12.baudrate = 2400
CT12.parity = 'E'
CT12.bytesize = 7
CT12.stopbits = 1
CT12.timeout = 3

logStart = datetime.now()
dtg = datetime.strftime(logStart, '%Y-%m-%d %H:%M:%S ')
ctlA = unichr(1)
bom = unichr(2)
eom = unichr(3)
bel = unichr(7)
CT12Name = [ctlA, 'CT12-NWC-test']
CT12Header = ['-Ceilometer Logfile \r\n', '-File created: ', dtg, '\r\n']

def write_ceilo ( text ) :
    f = open ('/data/CT12.log', 'a')
    f.write (text)
    f.close ()

write_ceilo(''.join(CT12Header))

CT12.open()

discard = CT12.readlines()
#print (discard)

while CT12.isOpen():
    response = CT12.readline()
    if len(response) >= 3:
        if response[0] == '\x02' :
            now=datetime.now()
            dtg=datetime.strftime(now, '-%Y-%m-%d %H:%M:%S\r\n')
            write_ceilo(dtg)
            write_ceilo(''.join(CT12Name))
            write_ceilo(response)

What can I do to make this rotate automatically, affixing either a date of rotation, or a serial number, for identification. I'm not looking to rotate any of these out, just keep a daily log file of the data. (or maybe an hourly file?)

Answer 1

For anyone arriving via Google, please don't move the log file out from under the logger while it is in use by calling the system copy of move commands or etc.

What you are looking for is a TimedRotatingFileHandler:

import time

import logging
from logging.handlers import TimedRotatingFileHandler

# format the log entries
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s %(message)s')

handler = TimedRotatingFileHandler('/path/to/logfile.log', 
                                   when='midnight',
                                   backupCount=10)
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)

# generate example messages
for i in range(10000):
    time.sleep(1)
    logger.debug('debug message')
    logger.info('informational message')
    logger.warn('warning')
    logger.error('error message')
    logger.critical('critical failure')

Answer 2

You can simply do this:

import os
import time
date1 = time.strftime('%Y%m%d%H%M%S')
cmd1= "cp logfile logfile{0}".format(date1)
cmd2= "cat /dev/null > logfile"
os.system(cmd1)
os.system(cmd2)

'logfile' is the name of the file. I have copied the older log a new log file with a name based on time and date and then emptied the original file. If you want to rotate it every one hour, put this script in cron.

Answer 3

For anyone who does not like the idea of rotating files, but simply wants to use a file handler that immediately writes to a file with a specific date in its name: it is not hard to write your own handler. Here is an example:

class FileHandlerWithOneFilePerPeriod(FileHandler):
"""A handler which writes formatted logging records to files, one file per period."""

def __init__(self, filename_pattern, mode='a', encoding=None, delay=False):
    """
    Constructs the file handler.

    :param filename_pattern: the filename. Use strftime() directives to specify the format of the period.
    For example, %Y%m%d can be used to generate one log file per day.
    :param mode: the mode to open the file before writing. Common values are 'w' for writing (truncating the file
    if it already exists), 'x' for creating and writing to a new file, and 'a' for appending (which on some Unix
    systems, means that all writes append to the end of the file regardless of the current seek position).
    :param encoding: encoding is the name of the encoding used to decode or encode the file. This should only be
    used in text mode.
    :param delay: True if the file is opened when the first log message is emitted; False if the file is opened now
    by the constructor.
    """
    self.filename_pattern = filename_pattern
    filename = datetime.now().strftime(self.filename_pattern)
    super().__init__(filename, mode, encoding, delay)

def emit(self, record: LogRecord):
    new_filename = datetime.fromtimestamp(record.created).strftime(self.filename_pattern)
    if self.stream is None:
        self.set_new_filename(new_filename)
    elif self.differs_from_current_filename(new_filename):
        self.close()
        self.set_new_filename(new_filename)

    super().emit(record)

def set_new_filename(self, new_filename):
    self.baseFilename = new_filename

def differs_from_current_filename(self, filename: str) -> bool:
    return filename != self.baseFilename

To use this handler, configure it with the following values in a dictionary (using logging.config.dictConfig():

version: 1
formatters:
  simple:
    format: '%(asctime)s %(name)s %(levelname)s %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout
  file:
    class: my_package.my_module.FileHandlerWithOneFilePerPeriod
    level: DEBUG
    formatter: simple
    filename_pattern: my_logging-%Y%m%d.log
root:
  level: DEBUG
  handlers: [console, file]

This will log to console and to file. One file per day is used. Change my_package and my_module to match the module where you put the handler. Change my_logging to a more appropriate name.

By changing the date pattern in filename_pattern you actually control when new files are created. Each time the pattern applied to the datetime a log message is created differs from the previous applied pattern, a new file will be created.