daemontools is a collection of tools for managing UNIX services.
Daemontools expects a collection of directories in
/service. Each directory contains a
run file, which must be executable. This file sets up the environment and runs the daemon. Note that all daemons will be started as root, since
supervise runs as
root. Each service directory may also contain a
log/ directory, which itself contains a
run, to run a separate logging program. I’ll have more to say about logging in the Logging stdout / stderr section.
run file is whatever you want it to be: a Bash script, a Ruby script, a Scala script, a compiled binary. Daemontools itself doesn’t care what it is exactly, as long as it’s executable. The simplest possible
1 #!/bin/sh 2 /opt/service/bin/daemon > /var/log/daemon.log
This has a number of problems, which I’ll solve later on.
I prefer to download and install
daemontools, rather than using the Ubuntu provided version. I prefer to use
/service rather than
/etc/service. Beyond that, I have no other reasons. Maybe I should stick with my distro-provided version. YMMV.
One note: on recent Ubuntus, I have to patch daemontools according to Installing Bernstein’s daemontools on CentOS 5. The problem is a missing include.
Once you have daemontools compiled and installed, you still have to start
svcscanboot somehow, and I’ve done so using
/etc/init/svscan.conf (Ubuntu 11.04, using Upstart):
1 start on runlevel  2 respawn 3 exec /command/svscanboot
Running a daemon as a service
To make things more concrete, I’ll setup carbon-cache to run under daemontools control. Carbon-cache is the daemon which accepts and writes data for the Graphite realtime graphing library.
1 #!/bin/bash 2 PATH=/usr/local/bin:/usr/bin 3 cd /opt/graphite/ 4 exec setuidgid graphite /usr/bin/python bin/carbon-cache.py --debug start
The first line indicates this is a Bash script. The second sets up a minimal environment for running this script. The third ensures we’re in the right directory. The fourth line is where the meat is, so let’s take it apart:
exec means to replace the Bash executable with whatever occurs next. It’s the same as exec(3). I do this in order to be able to signal the underlying process myself, rather than having to jump through hoops to signal the daemon;
setuidgid graphite is the way to drop privileges. The daemon itself doesn’t need to know how to drop privileges: when it boots, it will already be running using the correct privileges;
/usr/bin/python bin/carbon-cache.py --debug start is how I start the daemon itself.
--debug option to
carbon-cache is so it doesn’t daemonize itself: I want to keep it running in the foreground, again to be able to signal the process.
The service directory
It is imperative the service directory (the one which contains
run) retain its content on deploy. If you had a Rails application which you deployed using Capistrano, and you kept your service directory under version control (which you totally should), and you symlinked
/u/apps/seevibes/current/config/env, then each deploy will wipe
supervise‘s metadata: the contents of
/service/seevibes would be the contents of the new checkout, rather than the old one. If you had told
supervise to keep the daemon down then on deploy, the
down file won’t exist anymore: it would exist in the old release directory, rather than the new one. Instead, here’s how I replace my
run script on deploy:
1 cp /u/apps/seevibes/current/config/env/run /service/seevibes/run.new 2 mv -f /service/seevibes/run.new /service/seevibes/run # atomically replace run 3 svc -t /service/seevibes
run script itself is:
1 #!/bin/bash 2 PATH=/usr/local/bin:/usr/bin:/bin 3 RAILS_ROOT=`readlink -f /u/apps/seevibes/current` 4 cd $RAILS_ROOT 5 exec setuidgid deploy bundle exec thin start
Notice I’m calling
readlink to get an absolute path to the application’s root, with no symlinks. This ensures that the environment in which the daemon runs won’t suddenly change from under it’s feet.
Of course, these shenanigans are only necessary if the
run file itself has changed. If it hasn’t, a simple
svc -t /service/seevibes is enough to restart the daemon (
-t sends a
TERM signal; send other signals using the docs at
svc). But since automation makes everything much easier, why not add the appropriate task to your deployment script?
Setting up an environment for the daemon
If you need to setup environment variables, you use envdir.
envdir simply points to a directory containing a collection of files named after environment variables. The contents of the files indicates what the variable’s value will be set to, before
envdir executes the rest of it’s command line.
For example, here’s how I’d setup a JRuby application to run under 1.9 mode:
1 #!/bin/bash 2 PATH=/usr/local/bin:/usr/bin:/bin 3 cd /u/apps/seevibes/current 4 exec \ 5 envdir /etc/jruby/env \ 6 envdir /etc/seevibes/env \ 7 envdir /u/apps/seevibes/current/config/env \ 8 setuidgid deploy \ 9 bundle exec thin start
/u/apps/seevibes/current/config/env are directories that may or may not contain files that setup the environment for the daemon. In this instance, I have a file named
/etc/seevibes/env/JRUBY_OPTS whose sole contents is
--1.9 (with or without newline, because
envdir will take care of it). In the final process environment, the JRUBY_OPTS environment variable will exist, with the correct value. I have three calls to
envdir to enable a form of configuration inheritance.
See the Secure your environment variables section in the best practices document to secure the passwords you put in there.
Logging your daemon’s output
In the simplest possible log above, I simply redirect STDOUT to a file, and call it a day. It works, but has a number of issues:
- You lose STDERR;
- The log file isn’t necessarily rotated and may use up all your diskspace.
Daemontools supports a logger in the form of a separate daemon that you put in
/service/DAEMON/log. The same principles apply:
setuidgid, etc. Daemontools itself comes with a tool called
multilog which is designed to not lose data when it receives a TERM signal.
multilog expects as arguments a kind of script which tells it how to log the lines, and where to put them.
For example, you can use:
1 #!/bin/bash 2 PATH=/usr/local/bin:/usr/bin:/bin 3 4 mkdir -p /var/log/daemon 5 chown deploy:deploy /var/log/daemon 6 exec setuidgid deploy multilog s10485760 -n5 /var/log/daemon
Which will log output to
/var/log/daemon/current 10 MiB at a time, keeping up to 5 old log files. I also ensure the daemon’s output directory will exist and is writable at the same time.
See the Logging STDOUT / STDERR safely section of the best practices document for help on logging STDERR.
If you want, continue with Daemontools: Best Practices.