python logging performance comparison and options

Question

I am researching high performance logging in Python and so far have been disappointed by the performance of the python standard logging module - but there seem to be no alte

Answer 1

Python is not truly multi-threaded in a traditional sense. Whenever a thread is executing it has to own the gil (global interpreter lock). "threads" yield whenever they call into the system or have to wait on IO. This allows the interpreter thread to run other python "threads". This equates to asynchronous I/O.

Regardless of if the result of the logging message is used or dropped all of the work to evaluate the arguments for the logging message is done. As mentioned in other responses. However what is missed (and where the multi-threaded part of you question comes in) is that while writing a large amount to disk may be slow since modern computers have many cores the process of writing the output to the file will be farmed out to another core while the interpreter moves on to another python "thread". The operating system will complete the async disk write and little to no time will be lost to the disk write.

As long as the interpreter always has another thread to switch to virtually no time will be lost to the writes. The interpreter will only actually lose time if all python "threads" are blocked on I/O. Which is likely the case unless you are really swamping your disk.

Answer 2

The stdlib logging package provides a lot of flexibility and functionality for developers / devops / support staff, and that flexibility comes at some cost, obviously. If the need for performance trumps the need for flexibility, you need to go with something else. Did you take the steps to optimise described in the docs? A typical logging call takes of the order of tens of microseconds on reasonable hardware, which hardly seems excessive. However, logging in tight loops is seldom advisable, if only because the amount of info generated might take too much time to wade through.

The code to find the caller can be quite expensive, but is needed if you want e.g. filename and line number where the logging call was made.

QueueHandler is intended for scenarios where the logging I/O will take significant time and can't be done in-band. For example, a web application whose logs need to be sent by email to site administrators cannot risk using SMTPHandler directly, because the email handshake can be slow.

Don't forget that thread context switching in Python is slow. Did you try SocketHandler? There is a suitable starting point in the docs for a separate receiver process that does the actual I/O to file, email etc. So your process is only doing socket I/O and not doing context switches just for logging. And using domain sockets or UDP might be faster still, though the latter is of course lossy.

There are other ways to optimise. For example, standard handlers in logging do locking around emit(), for thread safety - if in a specific scenario under your control there is no contention for the handler, you could have a handler subclass that no-ops the lock acquisition and release. And so on.

Answer 3

If you want a better answer try to describe your problem in more detail, why you need such a huge number of messages to log? Logging was designed to record important information, especially warnings and errors, not every line you execute.

If logging takes more than 1% of your processing time, probably you are using it wrongly and that's not logging fault.

Second, related to performance: do not build the message before sending it to logging module (replace format % params with format command params). This is because logging does this for you, but much faster.