How to limit log file size in python

Question

I am using windows 7 and python 2.7. I want to limit my log file size to 5MB. My app, when it starts, writes to log file, and then the app terminates. When my app starts again, it will write in same log file. So app is not continuously running. App initiates, processes and terminates.

My code for logging is:

import logging
import logging.handlers
logging.basicConfig(filename=logfile.log, level="info", format='%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')
logging.info("*************************************************")

I tried with RotatingFileHandler but it didn't work

logging.handlers.RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024, backupCount=2, encoding=None, delay=0)

So, how can I enforce a file size limit in python?

Answer 1

Lose basicConfig and try this:

import logging
from logging.handlers import RotatingFileHandler

log_formatter = logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')

logFile = 'C:\\Temp\\log'

my_handler = RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024, 
                                 backupCount=2, encoding=None, delay=0)
my_handler.setFormatter(log_formatter)
my_handler.setLevel(logging.INFO)

app_log = logging.getLogger('root')
app_log.setLevel(logging.INFO)

app_log.addHandler(my_handler)

while True:
    app_log.info("data")

This works on my machine

Answer 2

When you use logging.basicConfig with a file, the log is attached with a file handler to handle writing to the file. afterwards you created another file handler to the same file with logging.handlers.RotatingFileHandler

Now, once a rotate is needed, RotatingFileHandler is trying to remove the old file but it can't becuase there is an open file handler

this can be seen if you look directly at the log file handlers -

import logging
from logging.handlers import RotatingFileHandler

log_name = 'c:\\log.log'
logging.basicConfig(filename=log_name)
log = logging.getLogger()
handler = RotatingFileHandler(name,maxBytes=1024,backupCount=1)
log.addHandler(handler)


[<logging.FileHandler object at 0x02AB9B50>, <logging.handlers.RotatingFileHandler object at 0x02AC1D90>]