daemontools is a collection of tools for managing UNIX services.

daemontools home page

Daemontools expects a collection of directories in /service. Each directory contains a run file, which must be executable. This file sets up the environment and runs the daemon. Note that all daemons will be started as root, since supervise runs as root. Each service directory may also contain a log/ directory, which itself contains a run, to run a separate logging program. I’ll have more to say about logging in the Logging stdout / stderr section.

The run file is whatever you want it to be: a Bash script, a Ruby script, a Scala script, a compiled binary. Daemontools itself doesn’t care what it is exactly, as long as it’s executable. The simplest possible run is:

1 #!/bin/sh
2 /opt/service/bin/daemon > /var/log/daemon.log

This has a number of problems, which I’ll solve later on.

Installation

I prefer to download and install daemontools, rather than using the Ubuntu provided version. I prefer to use /service rather than /etc/service. Beyond that, I have no other reasons. Maybe I should stick with my distro-provided version. YMMV.

One note: on recent Ubuntus, I have to patch daemontools according to Installing Bernstein’s daemontools on CentOS 5. The problem is a missing include.

Once you have daemontools compiled and installed, you still have to start svcscanboot somehow, and I’ve done so using /etc/init/svscan.conf (Ubuntu 11.04, using Upstart):

1 start on runlevel [12345]
2 respawn
3 exec /command/svscanboot

Running a daemon as a service

To make things more concrete, I’ll setup carbon-cache to run under daemontools control. Carbon-cache is the daemon which accepts and writes data for the Graphite realtime graphing library.

1 #!/bin/bash
2 PATH=/usr/local/bin:/usr/bin
3 cd /opt/graphite/
4 exec setuidgid graphite /usr/bin/python bin/carbon-cache.py --debug start

The first line indicates this is a Bash script. The second sets up a minimal environment for running this script. The third ensures we’re in the right directory. The fourth line is where the meat is, so let’s take it apart:

1. exec means to replace the Bash executable with whatever occurs next. It’s the same as exec(3). I do this in order to be able to signal the underlying process myself, rather than having to jump through hoops to signal the daemon;
2. setuidgid graphite is the way to drop privileges. The daemon itself doesn’t need to know how to drop privileges: when it boots, it will already be running using the correct privileges;
3. /usr/bin/python bin/carbon-cache.py --debug start is how I start the daemon itself.

The --debug option to carbon-cache is so it doesn’t daemonize itself: I want to keep it running in the foreground, again to be able to signal the process.

The service directory

It is imperative the service directory (the one which contains run) retain its content on deploy. If you had a Rails application which you deployed using Capistrano, and you kept your service directory under version control (which you totally should), and you symlinked /service/seevibes to /u/apps/seevibes/current/config/env, then each deploy will wipe supervise‘s metadata: the contents of /service/seevibes would be the contents of the new checkout, rather than the old one. If you had told supervise to keep the daemon down then on deploy, the down file won’t exist anymore: it would exist in the old release directory, rather than the new one. Instead, here’s how I replace my run script on deploy:

1 cp /u/apps/seevibes/current/config/env/run /service/seevibes/run.new
2 mv -f /service/seevibes/run.new /service/seevibes/run # atomically replace run
3 svc -t /service/seevibes

And the run script itself is:

1 #!/bin/bash
2 PATH=/usr/local/bin:/usr/bin:/bin
3 RAILS_ROOT=`readlink -f /u/apps/seevibes/current`
4 cd $RAILS_ROOT
5 exec setuidgid deploy bundle exec thin start

Notice I’m calling readlink to get an absolute path to the application’s root, with no symlinks. This ensures that the environment in which the daemon runs won’t suddenly change from under it’s feet.

Of course, these shenanigans are only necessary if the run file itself has changed. If it hasn’t, a simple svc -t /service/seevibes is enough to restart the daemon (-t sends a TERM signal; send other signals using the docs at svc). But since automation makes everything much easier, why not add the appropriate task to your deployment script?

Setting up an environment for the daemon

If you need to setup environment variables, you use envdir. envdir simply points to a directory containing a collection of files named after environment variables. The contents of the files indicates what the variable’s value will be set to, before envdir executes the rest of it’s command line.

For example, here’s how I’d setup a JRuby application to run under 1.9 mode:

 1 #!/bin/bash
 2 PATH=/usr/local/bin:/usr/bin:/bin
 3 cd /u/apps/seevibes/current
 4 exec                                         \
 5   envdir /etc/jruby/env                      \
 6   envdir /etc/seevibes/env                   \
 7   envdir /u/apps/seevibes/current/config/env \
 8   setuidgid deploy                           \
 9   bundle exec thin start

Where /etc/jruby/env, /etc/seevibes/env and /u/apps/seevibes/current/config/env are directories that may or may not contain files that setup the environment for the daemon. In this instance, I have a file named /etc/seevibes/env/JRUBY_OPTS whose sole contents is --1.9 (with or without newline, because envdir will take care of it). In the final process environment, the JRUBY_OPTS environment variable will exist, with the correct value. I have three calls to envdir to enable a form of configuration inheritance.

See the Secure your environment variables section in the best practices document to secure the passwords you put in there.

Logging your daemon’s output

In the simplest possible log above, I simply redirect STDOUT to a file, and call it a day. It works, but has a number of issues:

  • You lose STDERR;
  • The log file isn’t necessarily rotated and may use up all your diskspace.

Daemontools supports a logger in the form of a separate daemon that you put in /service/DAEMON/log. The same principles apply: envdir, setuidgid, etc. Daemontools itself comes with a tool called multilog which is designed to not lose data when it receives a TERM signal. multilog expects as arguments a kind of script which tells it how to log the lines, and where to put them.

For example, you can use:

1 #!/bin/bash
2 PATH=/usr/local/bin:/usr/bin:/bin
3 
4 mkdir -p /var/log/daemon
5 chown deploy:deploy /var/log/daemon
6 exec setuidgid deploy multilog s10485760 -n5 /var/log/daemon

Which will log output to /var/log/daemon/current 10 MiB at a time, keeping up to 5 old log files. I also ensure the daemon’s output directory will exist and is writable at the same time.

See the Logging STDOUT / STDERR safely section of the best practices document for help on logging STDERR.

If you want, continue with Daemontools: Best Practices.

Search

Your Host

A picture of me

I am François Beausoleil, a Ruby on Rails and Scala developer. During the day, I work on Seevibes, a platform to measure social interactions related to TV shows. At night, I am interested many things. Read my biography.

Top Tags

Books I read and recommend

Links

Projects I work on

Projects I worked on