Showing the right funcName when wrapping logger functionality in a custom class

Question

This is the formatting string that I am using for logging:

\'%(asctime)s - %(levelname)-10s - %(funcName)s - %(message)s\'

But to show the logg

Answer 1

Thanks to @glglgl I could come up with ad advanced findCaller

Please note the initialization of _logging_srcfile and _this_srcfile - inspired from the python logging source code

Of course you can put your own rules in the findCaller() - here i'm just excluding everything from the file where the custom logger is, EXCEPT the test_logging function.

IMPORTANT the custom logger is only retrieved when passing a name to the getLogger(name) factory. If you simply do logging.getLogger() you will get the RootLogger which is NOT your logger.

import sys
import os
import logging
# from inspect import currentframe
currentframe = lambda: sys._getframe(3)
_logging_srcfile = os.path.normcase(logging.addLevelName.__code__.co_filename)
_this_srcfile = __file__


def test_logging():
    logger = logging.getLogger('test')
    handler = logging.StreamHandler(sys.stderr)
    handler.setFormatter(logging.Formatter('%(funcName)s: %(message)s'))
    handler.setLevel(11)
    logger.addHandler(handler)
    logger.debug('Will not print')
    logger.your_function('Test Me')


class CustomLogger(logging.getLoggerClass()):
    def __init__(self, name, level=logging.NOTSET):
        super(CustomLogger, self).__init__(name, level)

    def your_function(self, msg, *args, **kwargs):
        # whatever you want to do here...
        self._log(12, msg, args, **kwargs)

    def findCaller(self):
        """
        Find the stack frame of the caller so that we can note the source
        file name, line number and function name.

        This function comes straight from the original python one
        """
        f = currentframe()
        # On some versions of IronPython, currentframe() returns None if
        # IronPython isn't run with -X:Frames.
        if f is not None:
            f = f.f_back
        rv = "(unknown file)", 0, "(unknown function)"
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            ## original condition
            # if filename == _logging_srcfile:
            ## PUT HERE YOUR CUSTOM CONDITION, eg:
            ## skip also this file, except the test_logging method which is used for debug
            if co.co_name != 'test_logging' and filename in [_logging_srcfile, _this_srcfile]:
                f = f.f_back
                continue
            rv = (co.co_filename, f.f_lineno, co.co_name)
            break
        return rv

logging.setLoggerClass(CustomLogger)

Answer 2

Someone has given the right answer. I will make a summary.

logging.Logger.findCaller(), it filter stack frames by logging._srcfile in original logging package.

So we do the same thing, filter our own logger wrapper my_log_module._srcfile. We replace the method logging.Logger.findCaller() of your logger instance dynamically.

BTW, please don't create a subclass of logging.Logger, logging package has no design for OOP when findCaller, pitty...yes?

# file: my_log_module.py, Python-2.7, define your logging wrapper here
import sys
import os
import logging
my_logger = logging.getLogger('my_log')

if hasattr(sys, '_getframe'): currentframe = lambda: sys._getframe(3)
# done filching

#
# _srcfile is used when walking the stack to check when we've got the first
# caller stack frame.
#
_srcfile = os.path.normcase(currentframe.__code__.co_filename)

def findCallerPatch(self):
    """
    Find the stack frame of the caller so that we can note the source
    file name, line number and function name.
    """
    f = currentframe()
    #On some versions of IronPython, currentframe() returns None if
    #IronPython isn't run with -X:Frames.
    if f is not None:
        f = f.f_back
    rv = "(unknown file)", 0, "(unknown function)"
    while hasattr(f, "f_code"):
        co = f.f_code
        filename = os.path.normcase(co.co_filename)
        if filename == _srcfile:
            f = f.f_back
            continue
        rv = (co.co_filename, f.f_lineno, co.co_name)
        break
    return rv

# DO patch
my_logger.findCaller = findCallerPatch

Ok, all ready. You can use your logger in other modules now, add your logging message format: lineno, path, method name, blablabla

# file: app.py
from my_log_module import my_logger
my_logger.debug('I can check right caller now')

Or you can use a elegant way, but don't use global logging.setLoggerClass

# file: my_log_modue.py
import logging
my_logger = logging.getLogger('my_log')

class MyLogger(logging.Logger):
    ...

my_logger.__class__ = MyLogger

Answer 3

Essentially, the code to blame lies in the Logger class:

This method

def findCaller(self):
    """
    Find the stack frame of the caller so that we can note the source
    file name, line number and function name.
    """
    f = currentframe()
    #On some versions of IronPython, currentframe() returns None if
    #IronPython isn't run with -X:Frames.
    if f is not None:
        f = f.f_back
    rv = "(unknown file)", 0, "(unknown function)"
    while hasattr(f, "f_code"):
        co = f.f_code
        filename = os.path.normcase(co.co_filename)
        if filename == _srcfile:
            f = f.f_back
            continue
        rv = (co.co_filename, f.f_lineno, co.co_name)
        break
    return rv

returns the first function in the chain of callers which doesn't belong to the current module.

You could subclass Logger and override this method by adding a slightly more complex logic. skipping another level of calling depth or adding another condition.

---

In your very special case, it would probably be simpler to refrain from the automatic line splitting and to do

logger.progress('Hello %s', name)
logger.progress('How are you doing?')

or to do

def splitter(txt, *args)
    txt = txt % (args)
    for line in txt.split('\n'):
        yield line

for line in splitter('Hello %s\nHow are you doing?', name):
    logger.progress(line)

and have a

def progress(self, txt, *args):
    self.log(self.PROGRESS, txt, *args)

Probably it will save you a lot of headache.

EDIT 2: No, that won't help. It now would show you progress as your caller function name...

Answer 4

First of all according to your code it's clear why it happens, levelname and funcName "belongs" to self.log so when you call to self.log(level, line) the levelname is level and funcName is line.

You have 2 options IMHO:

1. To use inspect module to get the current method and to deliver it inside the message, then you can parse it and to use it very easily.

2. A better approach will be to use inspect inside split_line to get the "father" method you can change the number(3) in the following code to "play" with the hierarchy of the methods.

example of using inspect to get current method

from inspect import stack

class Foo:
    def __init__(self):
        print stack()[0][3]

f = Foo()

Answer 5

This is fixed in Python 3.8 with addition of the stacklevel param. However, I took the current implementation of findCaller from cpython to make a Python 3.7 compatible version.

Taken from a combination of the answers above:

import sys,os

#Get both logger's and this file's path so the wrapped logger can tell when its looking at the code stack outside of this file.
_loggingfile = os.path.normcase(logging.__file__)
if hasattr(sys, 'frozen'): #support for py2exe
    _srcfile = "logging%s__init__%s" % (os.sep, __file__[-4:])
elif __file__[-4:].lower() in ['.pyc', '.pyo']:
    _srcfile = __file__[:-4] + '.py'
else:
    _srcfile = __file__
_srcfile = os.path.normcase(_srcfile)
_wrongCallerFiles = set([_loggingfile, _srcfile])

#Subclass the original logger and overwrite findCaller
class WrappedLogger(logging.Logger):
    def __init__(self, name):
        logging.Logger.__init__(self, name)

    #Modified slightly from cpython's implementation https://github.com/python/cpython/blob/master/Lib/logging/__init__.py#L1374
    def findCaller(self, stack_info=False, stacklevel=1):
        """
        Find the stack frame of the caller so that we can note the source
        file name, line number and function name.
        """
        f = currentframe()
        #On some versions of IronPython, currentframe() returns None if
        #IronPython isn't run with -X:Frames.
        if f is not None:
            f = f.f_back
        orig_f = f
        while f and stacklevel > 1:
            f = f.f_back
            stacklevel -= 1
        if not f:
            f = orig_f
        rv = "(unknown file)", 0, "(unknown function)", None
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            if filename in _wrongCallerFiles:
                f = f.f_back
                continue
            sinfo = None
            if stack_info:
                sio = io.StringIO()
                sio.write('Stack (most recent call last):\n')
                traceback.print_stack(f, file=sio)
                sinfo = sio.getvalue()
                if sinfo[-1] == '\n':
                sinfo = sinfo[:-1]
            sio.close()
        rv = (co.co_filename, f.f_lineno, co.co_name, sinfo)
        break
    return rv

Answer 6

You can merge progress method and split_line method:

def progress(self, txt, *args, **kwargs):
    if self.isEnabledFor(self.PROGRESS):
        txt = txt % (args)
        for line in txt.split('\n'):
            self._log(self.PROGRESS, line, [], **kwargs)