O'Reilly logo

Linux Server Hacks by Rob Flickenger

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Hack #4. Creating a Persistent Daemon with init

Make sure that your process stays up, no matter what

There are a number of scripts that will automatically restart a process if it exits unexpectedly. Perhaps the simplest is something like:

$ while : ; do echo "Run some code here..."; sleep 1; done

If you run a foreground process in place of that echo line, then the process is always guaranteed to be running (or, at least, it will try to run). The : simply makes the while always execute (and is more efficient than running /bin/true, as it doesn't have to spawn an external command on each iteration). Definitely do not run a background process in place of the echo, unless you enjoy filling up your process table (as the while will then spawn your command as many times as it can, one every second). But as far as cool hacks go, the while approach is fairly lacking in functionality.

What happens if your command runs into an abnormal condition? If it exits immediately, then it will retry every second, without giving any indication that there is a problem (unless the process has its own logging system or uses syslog). It might make sense to have something watch the process, and stop trying to respawn it if it returns too quickly after a few tries.

There is a utility already present on every Linux system that will do this automatically for you: init . The same program that brings up the system and sets up your terminals is perfectly suited for making sure that programs are always running. In fact, that is its primary job.

You can add arbitrary lines to /etc/inittab specifying programs you'd like init to watch for you:

zz:12345:respawn:/usr/local/sbin/my_daemon

The inittab line consists of an arbitrary (but unique) two character identification string (in this case, zz), followed by the runlevels that this program should be run in, then the respawn keyword, and finally the full path to the command. In the above example, as long as my_daemon is configured to run in the foreground, init will respawn another copy whenever it exits. After making changes to inittab, be sure to send a HUP to init so it will reload its configuration. One quick way to do this is:

# kill -HUP 1

If the command respawns too quickly, then init will postpone execution for a while, to keep it from tying up too many resources. For example:

zz:12345:respawn:/bin/touch /tmp/timestamp

This will cause the file /tmp/timestamp to be touched several times a second, until init decides that enough is enough. You should see this message in /var/log/messages almost immediately:

Sep 8 11:28:23 catlin init: Id "zz" respawning too fast: disabled for 5 minutes

In five minutes, init will try to run the command again, and if it is still respawning too quickly, it will disable it again.

Obviously, this method is fine for commands that need to run as root, but what if you want your auto-respawning process to run as some other user? That's no problem: use sudo:

zz:12345:respawn:/usr/bin/sudo -u rob /bin/touch /tmp/timestamp

Now that touch will run as rob, not as root. If you're trying these commands as you read this, be sure to remove the existing /tmp/timestamp before trying this sudo line. After sending a HUP to init, take a look at the timestamp file:

rob@catlin:~# ls -al /tmp/timestamp 
-rw-r--r-- 1 rob users 0 Sep 8 11:28 /tmp/timestamp

The two drawbacks to using init to run arbitrary daemons are that you need to comment out the line in inittab if you need to bring the daemon down (since it will just respawn if you kill it) and that only root can add entries to inittab. But for keeping a process running that simply must stay up no matter what, init does a great job.

See also:

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required