Stopping gearman workers nicely

Question

I have a number of Gearman workers running constantly, saving things like records of user page views, etc. Occasionally, I\'ll update the PHP code that is used by the Gearman w

Answer 1

http://phpscaling.com/2009/06/23/doing-the-work-elsewhere-sidebar-running-the-worker/

Like the above article demonstrates, I've run a worker inside a BASH shell script, exiting occasionally between jobs to cleanup (or re-load the worker-script) - or if a given task is given to it it can exit with a specific exit code and to shut down.

Answer 2

What I do is use gearmadmin to check if there are any jobs running. I used the admin API to make a UI for this. When the jobs are sitting idly, there is no harm in killing them.

Answer 3

Solution 1

---

Generally I run my workers with the unix daemon utility with the -r flag and let them expire after one job. Your script will end gracefully after each iteration and daemon will restart automatically.

Your workers will be stale for one job but that may not be as big a deal to you as losing data

This solution also has the advantage of freeing up memory. You may run into problems with memory if you're doing large jobs as PHP pre 5.3 has god awful GC.

Solution 2

---

You could also add a quit function to all of your workers that exits the script. When you'd like to restart you simply give gearman calls to quit with a high priority.

Answer 4

This would fit nicely into your continuous integration system. I hope you have it or you should have it soon :-)

As you check in new code, it automatically gets built and deployed onto the server. As a part of the build script, you kill all workers, and launch new ones.

Answer 5

If someone were looking for answer for a worker running perl, that's part of what the GearmanX::Starter library is for. You can stop workers after completing the current job two different ways: externally by sending the worker process a SIGTERM, or programmatically by setting a global variable.

Answer 6

I ran into this same problem and came up with a solution for python 2.7.

I'm writing a python script which uses gearman to communicate with other components on the system. The script will have multiple workers, and I have each worker running in separate thread. The workers all receive gearman data, they process and store that data on a message queue, and the main thread can pull the data off of the queue as necessary.

My solution to cleanly shutting down each worker was to subclass gearman.GearmanWorker and override the work() function:

from gearman import GearmanWorker
POLL_TIMEOUT_IN_SECONDS = 60.0
class StoppableWorker(GearmanWorker):
    def __init__(self, host_list=None):
        super(StoppableWorker,self).__init__(host_list=host_list)
        self._exit_runloop = False


    # OVERRIDDEN
    def work(self, poll_timeout=POLL_TIMEOUT_IN_SECONDS):
        worker_connections = []
        continue_working = True

        def continue_while_connections_alive(any_activity):
            return self.after_poll(any_activity)

        while continue_working and not self._exit_runloop:
            worker_connections = self.establish_worker_connections()
            continue_working = self.poll_connections_until_stopped(
                worker_connections,
                continue_while_connections_alive,
                timeout=poll_timeout)

        for current_connection in worker_connections:
            current_connection.close()

        self.shutdown()


    def stopwork(self):
        self._exit_runloop = True

Use it just like GearmanWorker. When it's time to exit the script, call the stopwork() function. It won't stop immediately--it can take up to poll_timeout seconds before it kicks out of the run loop.

There may be multiple smart ways to invoke the stopwork() function. In my case, I create a temporary gearman client in the main thread. For the worker that I'm trying to shutdown, I send a special STOP command through the gearman server. When the worker gets this message, it knows to shut itself down.

Hope this helps!