Python watchdog for another Python process technique?

Question

I have a realtime data grabber that runs indefinitely, grabbing data over HTTP and putting it in a MySQL database every few seconds.

In my program, I have a while True loop that spawns workers (functions that download the data and save it) whenever the last spawned time is greater than X seconds:

while True:
    if _last_updated - datetime.now() > timedelta(seconds=5):
        green_pool.spawn_n(worker) # yes I'm using Eventlet!
        _last_updated = datetime.now()

What would be the best way to ensure that this module always does work, never freezes and is never down? Should I be checking the green pool size? I was thinking about writing a watchdog for it in Python, would you recommend doing so? If so, what things should I keep in mind?

Best

Answer 1

It might be overkill, but I would look at using supervisord. It's a process for controlling other processes (somewhat like init.d). It will allow you to start/stop/restart your control script containing the while True: loop. It will also auto-restart the control script if it stops working.

As you mention, you should keep tabs on the pool size and the success/failure of worker spawning within your control script. But to ensure that the control script is always running, supervisord fits the bill.