How to process SIGTERM signal gracefully?

Question

Let\'s assume we have such a trivial daemon written in python:

def mainloop():
    while True:
        # 1. do
        # 2. some
        # 3. important
              


        

Answer 1

Based on the previous answers, I have created a context manager which protects from sigint and sigterm.

import logging
import signal
import sys


class TerminateProtected:
    """ Protect a piece of code from being killed by SIGINT or SIGTERM.
    It can still be killed by a force kill.

    Example:
        with TerminateProtected():
            run_func_1()
            run_func_2()

    Both functions will be executed even if a sigterm or sigkill has been received.
    """
    killed = False

    def _handler(self, signum, frame):
        logging.error("Received SIGINT or SIGTERM! Finishing this block, then exiting.")
        self.killed = True

    def __enter__(self):
        self.old_sigint = signal.signal(signal.SIGINT, self._handler)
        self.old_sigterm = signal.signal(signal.SIGTERM, self._handler)

    def __exit__(self, type, value, traceback):
        if self.killed:
            sys.exit(0)
        signal.signal(signal.SIGINT, self.old_sigint)
        signal.signal(signal.SIGTERM, self.old_sigterm)


if __name__ == '__main__':
    print("Try pressing ctrl+c while the sleep is running!")
    from time import sleep
    with TerminateProtected():
        sleep(10)
        print("Finished anyway!")
    print("This only prints if there was no sigint or sigterm")

Answer 2

The simplest solution I have found, taking inspiration by responses above is

class SignalHandler:

    def __init__(self):

        # register signal handlers
        signal.signal(signal.SIGINT, self.exit_gracefully)
        signal.signal(signal.SIGTERM, self.exit_gracefully)

        self.logger = Logger(level=ERROR)

    def exit_gracefully(self, signum, frame):
        self.logger.info('captured signal %d' % signum)
        traceback.print_stack(frame)

        ###### do your resources clean up here! ####

        raise(SystemExit)

Answer 3

Here is a simple example without threads or classes.

import signal

run = True

def handler_stop_signals(signum, frame):
    global run
    run = False

signal.signal(signal.SIGINT, handler_stop_signals)
signal.signal(signal.SIGTERM, handler_stop_signals)

while run:
    pass # do stuff including other IO stuff

Answer 4

First, I'm not certain that you need a second thread to set the shutdown_flag.
Why not set it directly in the SIGTERM handler?

An alternative is to raise an exception from the SIGTERM handler, which will be propagated up the stack. Assuming you've got proper exception handling (e.g. with with/contextmanager and try: ... finally: blocks) this should be a fairly graceful shutdown, similar to if you were to Ctrl+C your program.

Example program signals-test.py:

#!/usr/bin/python

from time import sleep
import signal
import sys


def sigterm_handler(_signo, _stack_frame):
    # Raises SystemExit(0):
    sys.exit(0)

if sys.argv[1] == "handle_signal":
    signal.signal(signal.SIGTERM, sigterm_handler)

try:
    print "Hello"
    i = 0
    while True:
        i += 1
        print "Iteration #%i" % i
        sleep(1)
finally:
    print "Goodbye"

Now see the Ctrl+C behaviour:

$ ./signals-test.py default
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
^CGoodbye
Traceback (most recent call last):
  File "./signals-test.py", line 21, in <module>
    sleep(1)
KeyboardInterrupt
$ echo $?
1

This time I send it SIGTERM after 4 iterations with kill $(ps aux | grep signals-test | awk '/python/ {print $2}'):

$ ./signals-test.py default
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
Terminated
$ echo $?
143

This time I enable my custom SIGTERM handler and send it SIGTERM:

$ ./signals-test.py handle_signal
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
Goodbye
$ echo $?
0

Answer 5

I think you are near to a possible solution.

Execute mainloop in a separate thread and extend it with the property shutdown_flag. The signal can be caught with signal.signal(signal.SIGTERM, handler) in the main thread (not in a separate thread). The signal handler should set shutdown_flag to True and wait for the thread to end with thread.join()

Answer 6

Found easiest way for me. Here an example with fork for clarity that this way is useful for flow control.

import signal
import time
import sys
import os

def handle_exit(sig, frame):
    raise(SystemExit)

def main():
    time.sleep(120)

signal.signal(signal.SIGTERM, handle_exit)

p = os.fork()
if p == 0:
    main()
    os._exit()

try:
    os.waitpid(p, 0)
except (KeyboardInterrupt, SystemExit):
    print('exit handled')
    os.kill(p, 15)
    os.waitpid(p, 0)