Stopping gearman workers nicely

Question

I have a number of Gearman workers running constantly, saving things like records of user page views, etc. Occasionally, I\'ll update the PHP code that is used by the Gearman w

Answer 1

I ran into this same problem and came up with a solution for python 2.7.

I'm writing a python script which uses gearman to communicate with other components on the system. The script will have multiple workers, and I have each worker running in separate thread. The workers all receive gearman data, they process and store that data on a message queue, and the main thread can pull the data off of the queue as necessary.

My solution to cleanly shutting down each worker was to subclass gearman.GearmanWorker and override the work() function:

from gearman import GearmanWorker
POLL_TIMEOUT_IN_SECONDS = 60.0
class StoppableWorker(GearmanWorker):
    def __init__(self, host_list=None):
        super(StoppableWorker,self).__init__(host_list=host_list)
        self._exit_runloop = False


    # OVERRIDDEN
    def work(self, poll_timeout=POLL_TIMEOUT_IN_SECONDS):
        worker_connections = []
        continue_working = True

        def continue_while_connections_alive(any_activity):
            return self.after_poll(any_activity)

        while continue_working and not self._exit_runloop:
            worker_connections = self.establish_worker_connections()
            continue_working = self.poll_connections_until_stopped(
                worker_connections,
                continue_while_connections_alive,
                timeout=poll_timeout)

        for current_connection in worker_connections:
            current_connection.close()

        self.shutdown()


    def stopwork(self):
        self._exit_runloop = True

Use it just like GearmanWorker. When it's time to exit the script, call the stopwork() function. It won't stop immediately--it can take up to poll_timeout seconds before it kicks out of the run loop.

There may be multiple smart ways to invoke the stopwork() function. In my case, I create a temporary gearman client in the main thread. For the worker that I'm trying to shutdown, I send a special STOP command through the gearman server. When the worker gets this message, it knows to shut itself down.

Hope this helps!