Python Daemon Packaging Best Practices

Question

I have a tool which I have written in python and generally should be run as a daemon. What are the best practices for packaging this tool for distribution, particularly how sho

Answer 1

To answer one part of your question, there are no tools I know of that will do daemon setup portably even across Linux systems let alone Windows or Mac OS X.

Most Linux distributions seem to be using start-stop-daemon within init scripts now, but you're still going to have minor difference in filesystem layout and big differences in packaging. Using autotools/configure, or distutils/easy_install if your project is all Python, will go a long way to making it easier to build packages for different Linux/BSD distributions.

Windows is a whole different game and will require Mark Hammond's win32 extensions and maybe Tim Golden's WMI extensions.

I don't know Launchd except that "none of the above" are relevant.

For tips on daemonizing Python scripts, I would look to Python apps that are actually doing it in the real world, for example inside Twisted.

Answer 2

"generally should be run as a daemon?"

Doesn't -- on surface -- make a lot of sense. "Generally" isn't sensible. It's either a a daemon or not. You might want to update your question.

For examples of daemons, read up on daemons like Apache's httpd or any database server (they're daemons) or the SMTPD mail daemon.

Or, perhaps, read up on something simpler, like the FTP daemon, SSH daemon, Telnet daemon.

In Linux world, you'll have your application installation directory, some working directory, plus the configuration file directories.

We use /opt/ourapp for the application (it's Python, but we don't install in Python's lib/site-packages)

We use /var/ourapp for working files and our configuration files.

We could use /etc/ourapp for configuration files -- it would be consistent -- but we don't.

We don't -- yet -- use the init.d scripts for startup. But that's the final piece, automated startup. For now, we have sys admins start the daemons.

This is based, partly, on http://www.pathname.com/fhs/ and http://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/Linux-Filesystem-Hierarchy.html.

Answer 3

The best tool I found for helping with init.d scripts is "start-stop-daemon". It will run any application, monitor run/pid files, create them when necessary, provide ways to stop the daemon, set process user/group ids, and can even background your process.

For example, this is a script which can start/stop a wsgi server:

#! /bin/bash

case "$1" in
  start)
    echo "Starting server"

    # Activate the virtual environment
    . /home/ali/wer-gcms/g-env/bin/activate

    # Run start-stop-daemon, the $DAEMON variable contains the path to the
    # application to run
    start-stop-daemon --start --pidfile $WSGI_PIDFILE \
        --user www-data --group www-data \
        --chuid www-data \
        --exec "$DAEMON"
    ;;
  stop)
    echo "Stopping WSGI Application"

    # Start-stop daemon can also stop the application by sending sig 15
    # (configurable) to the process id contained in the run/pid file
    start-stop-daemon --stop --pidfile $WSGI_PIDFILE --verbose
    ;;
  *)
    # Refuse to do other stuff
    echo "Usage: /etc/init.d/wsgi-application.sh {start|stop}"
    exit 1
    ;;
esac

exit 0

You can also see there an example of how to use it with a virtualenv, which I would always recommend.

Answer 4

Not a silver bullet for what you're asking, but check out supervisord. It handles all the fun bits of managing processes. I use it heavily in a large production environment. Also, it's written in Python!