How to pickle loggers?

Question

Working on a project that requires that I am able to pickle the container object at any point, since we expect it to fail on external conditions quite frequently and be able to

Answer 1

You could create a class that wraps the logger and implements __getstate__ and __setstate__.

This is pasted from http://docs.python.org/library/pickle.html. The fh is handled in a way which may be similar to what you need.

#!/usr/local/bin/python

class TextReader:
    """Print and number lines in a text file."""
    def __init__(self, file):
        self.file = file
        self.fh = open(file)
        self.lineno = 0

    def readline(self):
        self.lineno = self.lineno + 1
        line = self.fh.readline()
        if not line:
            return None
        if line.endswith("\n"):
            line = line[:-1]
        return "%d: %s" % (self.lineno, line)

    def __getstate__(self):
        odict = self.__dict__.copy() # copy the dict since we change it
        del odict['fh']              # remove filehandle entry
        return odict

    def __setstate__(self, dict):
        fh = open(dict['file'])      # reopen file
        count = dict['lineno']       # read from file...
        while count:                 # until line count is restored
            fh.readline()
            count = count - 1
        self.__dict__.update(dict)   # update attributes
        self.fh = fh                 # save the file object

Answer 2

New in Python 3.7 (bpo30520)

Logger can now be pickled like many other objects.

import pickle
import logging
log = logging.getLogger(__name__)
logger_pickle = pickle.dumps(log)

# and of coarse, to load:
log = pickle.loads(logger_pickle)

Answer 3

You could have used dill here, which can pickle loggers and locks.

>>> class foo:
...    def __init__(self):
...        self.logger = logging.getLogger("package.foo")
... 
>>> import dill
>>> import logging
>>> 
>>> f = foo()
>>> _f = dill.dumps(f)  
>>> f_ = dill.loads(_f)
>>> f_.logger
<logging.Logger object at 0x110b1d250>

Answer 4

Found a very similar question here, with an answer that worked for me:

How to stop attributes from being pickled in Python

edit: used this answer: How to stop attributes from being pickled in Python

Answer 5

You could also create a class that implements a property which returns the needed logger. Every class which inherits from this "LoggerMixin" is now able to use the logger in the same way you were using it before.

class LoggerMixin():
    @property
    def logger(self):
        component = "{}.{}".format(type(self).__module__, type(self).__name__)
        return logging.getLogger(component)

class Foo(LoggerMixin):
    def __init__(self):
        self.logger.info("initialize class")

    def bar(self):
        self.logger.info("execute bar")