Should I use Events, Semaphores, Locks, Conditions, or a combination thereof to manage safely exiting my multithreaded Python program?

Question

I am writing a multithreaded python program in which the main thread and the other threads it spawns run as daemons (but not with Thread.daemon=True) that look for certain files

Answer 1

I would use an Event to signal to a thread that it should exit:

• create an event in __init__
• use the event's wait() in run() for sleep and for checking when to exit
• set the event from outside to stop the thread

To handle exceptions within a thread, I would have a try/ except block around everything it does. When something is caught, store the exception (and/or any other info you need), clean up and exit the thread.

Outside, in the main thread, check for the store exceptions in all threads, if any exception is found, signal to all threads that they should exit.

To handle exceptions in the main thread (which includes also SIGINT), have a try/except block there and signal to all threads to stop.

All together, with dummy exceptions and debug prints:

import threading
import time

class MyThread(threading.Thread):
    def __init__(self):
        super().__init__()
        self.stop_requested = threading.Event()
        self.exception = None

    def run(self):
        try:
            # sleep for 1 second, or until stop is requested, stop if stop is requested
            while not self.stop_requested.wait(1):
                # do your thread thing here
                print('in thread {}'.format(self))

                # simulate a random exception:
                import random
                if random.randint(0, 50) == 42:
                    1 / 0
        except Exception as e:
            self.exception = e

        # clean up here
        print('clean up thread {}'.format(self))

    def stop(self):
        # set the event to signal stop
        self.stop_requested.set()

# create and start some threads
threads = [MyThread(), MyThread(), MyThread(), MyThread()]
for t in threads:
    t.start()

# main thread looks at the status of all threads
try:
    while True:
        for t in threads:
            if t.exception:
                # there was an error in a thread - raise it in main thread too
                # this will stop the loop
                raise t.exception
        time.sleep(0.2)

except Exception as e:
    # handle exceptions any way you like, or don't
    # This includes exceptions in main thread as well as those in other threads
    # (because of "raise t.exception" above)
    print(e)

finally:
    print('clan up everything')
    for t in threads:
        # threads will know how to clean up when stopped
        t.stop()