Python: multiprocessing.map: If one process raises an exception, why aren't other processes' finally blocks called?

Question

My understanding is that finally clauses must *always* be executed if the try has been entered.

import random         


        

Answer 1

Short answer: SIGTERM trumps finally.

Long answer: Turn on logging with mp.log_to_stderr():

import random
import multiprocessing as mp
import time
import logging

logger=mp.log_to_stderr(logging.DEBUG)

def Process(x):
    try:
        logger.info(x)
        time.sleep(random.random())
        raise Exception('Exception: ' + x)
    finally:
        logger.info('Finally: ' + x)

result=mp.Pool(3).map(Process, ['1','2','3'])

The logging output includes:

[DEBUG/MainProcess] terminating workers

Which corresponds to this code in multiprocessing.pool._terminate_pool:

    if pool and hasattr(pool[0], 'terminate'):
        debug('terminating workers')
        for p in pool:
            p.terminate()

Each p in pool is a multiprocessing.Process, and calling terminate (at least on non-Windows machines) calls SIGTERM:

from multiprocessing/forking.py:

class Popen(object)
    def terminate(self):
        ...
            try:
                os.kill(self.pid, signal.SIGTERM)
            except OSError, e:
                if self.wait(timeout=0.1) is None:
                    raise

So it comes down to what happens when a Python process in a try suite is sent a SIGTERM.

Consider the following example (test.py):

import time    
def worker():
    try:
        time.sleep(100)        
    finally:
        print('enter finally')
        time.sleep(2) 
        print('exit finally')    
worker()

If you run it, then send it a SIGTERM, then the process ends immediately, without entering the finally suite, as evidenced by no output, and no delay.

In one terminal:

% test.py

In second terminal:

% pkill -TERM -f "test.py"

Result in first terminal:

Terminated

Compare that with what happens when the process is sent a SIGINT (C-c):

In second terminal:

% pkill -INT -f "test.py"

Result in first terminal:

enter finally
exit finally
Traceback (most recent call last):
  File "/home/unutbu/pybin/test.py", line 14, in 
    worker()
  File "/home/unutbu/pybin/test.py", line 8, in worker
    time.sleep(100)        
KeyboardInterrupt

Conclusion: SIGTERM trumps finally.