Can structured logging be done with Pythons standard library?

Question

I recently read about structured logging (here). The idea seems to be to log not by appending simple strings as a line to a logfile, but instead JSON objects. This makes it possible to analyze the logfile by automatic tools.

Can Pythons logging library do structured logging? If not, is there a "mainstream" solution for it (e.g. like numpy/scipy is the mainstream solution for scientific calculations)? I found structlog, but I'm not sure how widespread it is.

Answer 1

Have you looked at python docs site section describing Implementing structured logging that explain how python built-in logger can be utilized for structured logging?

Below is a simple example as listed on above site .

import json
import logging

class StructuredMessage(object):
    def __init__(self, message, **kwargs):
        self.message = message
        self.kwargs = kwargs

    def __str__(self):
        return '%s >>> %s' % (self.message, json.dumps(self.kwargs))

m = StructuredMessage   # optional, to improve readability

logging.basicConfig(level=logging.INFO, format='%(message)s')
logging.info(m('message 1', foo='bar', bar='baz', num=123, fnum=123.456))

Which results in following log.

message 1 >>> {"fnum": 123.456, "num": 123, "bar": "baz", "foo": "bar"}

Hope this helps.

Answer 2

If you install python-json-logger (288 stars, 70 forks) and have a logging configuration (YAML) like the following, you will get a structured logging file.

version: 1
formatters:
    detailed:
        class: logging.Formatter
        format: '[%(asctime)s]:[%(levelname)s]: %(message)s'
    json:
        class: pythonjsonlogger.jsonlogger.JsonFormatter
        format: '%(asctime)s %(levelname)s %(message)s'
handlers:
    console:
        class: logging.StreamHandler
        level: INFO
        formatter: detailed
    file:
        class: logging.FileHandler
        filename: logfile.log
        level: DEBUG
        formatter: json
root:
    level: DEBUG
    handlers:
        - console
        - file

Exceptions

You might also want to make exceptions / tracebacks use the structured format.

See Can I make Python output exceptions in one line / via logging?