How can I inherit parent logger when using Python's multiprocessing? Especially for paramiko

Question

I\'m using Python\'s multiprocessing. I have set the logger in parent process, but I can\'t just simply inherit the parent\'s logging setting.

I don\'t worry about mixin

Answer 1

In Python, subprocesses are started on POSIX standard. SubProcesses in POSIX standard are created using fork system call. The child process created using fork is essentially a copy of everything in the parent process' memory. In your case, child process will have access to logger from parent.

Warning: fork copies everything; but, does not copy threads. Any threads running in parent process do not exist in child process.

import logging
from multiprocessing import Pool
from os import getpid

def runs_in_subprocess():
    logging.info(
        "I am the child, with PID {}".format(getpid()))

if __name__ == '__main__':
    logging.basicConfig(
        format='GADZOOKS %(message)s', level=logging.DEBUG)

    logging.info(
        "I am the parent, with PID {}".format(getpid()))

    with Pool() as pool:
        pool.apply(runs_in_subprocess)

Output:

GADZOOKS I am the parent, with PID 3884
GADZOOKS I am the child, with PID 3885

Notice how child processes in your pool inherit the parent process’ logging configuration

You might get into problem of deadlocks so beware of following:

1. Whenever the thread in the parent process writes a log messages, it adds it to a Queue. That involves acquiring a lock.

2. If the fork() happens at the wrong time, the lock is copied in an acquired state.

3. The child process copies the parent’s logging configuration—including the queue. Whenever the child process writes a log message, it tries to write it to the queue.

4. That means acquiring the lock, but the lock is already acquired.

5. The child process now waits for the lock to be released.

6. The lock will never be released, because the thread that would release it wasn’t copied over by the fork().

In python3, you could avoid this using get_context.

from multiprocessing import get_context

def your_func():
    with get_context("spawn").Pool() as pool:
        # ... everything else is unchanged

Advice:

1. Using get_context create a new Pool and use process' within this pool to do the job for you.

2. Every process from the pool will have access to the parent process' log config.