Killing a daemon using a PID file

Question

A common Linux/UNIX idiom when it comes to running daemons is to spawn the daemon, and create a PID file which just contains the process ID of the daemon. This way, to stop/restart the daemon you can simply have scripts which kill $(cat mydaemon.pid)

Now, there's a lot of opportunity here for inconsistent state. Suppose the machine running the daemon is forcefully shut off, then restarted. Now you have a PID file which refers to a non-existent process.

Okay, so no problem... your daemon will just try to kill the non-existent process, find that it's not a real process, and continue as usual.

But... what if it is a real process - just not your daemon? What if it's someone else's process, or some other important process? You have no way of knowing - so killing it is potentially dangerous. One possibility would be to check the name of the process. Of course, this isn't foolproof either because there's no reason another process might not have the same name. Especially, if for example, your daemon runs under an interpreter, like Python, in which case the process name will never be something unique - it will simply be "python", in which case you might inadvertently kill someone else's process.

So how can we handle a situation like this where we need to restart a daemon? How can we know the PID in the pid file necessarily is the daemon?

Answer 1

You just keep adding on layers of paranoia:

• pid file
• process name matching
• some communication channel/canary

The most important thing that you could to in order to ensure that the pid isn't stale following a reboot is to store it in /var/run , which is a location that is guaranteed to be cleared every reboot.

For process name matching, you can actually redefine the name of the process at the fork/exec point, which will allow you to use some unique name.

The communication channel/canary is a little more complex and is prone to some gotchas. If a daemon creates a named socket, then the presence of the socket + the ability to connect and communicate with the daemon would be considered evidence that the process is running.

Answer 2

If you really want to provide a script for your users, you could the let the daemon process manage its pidfile on its own and add an atexit and a SIGABRT handler to unlink the pidfile even on unclean shutdown.

More ways include also storing the process startup time in the pidfile. Together with volatile storage (eg /var/run ) this is a pretty reliable way to identify a process. This makes the kill command a bit more complicated though.

However, I personally think that a daemon developer should not care (too much) about this and let this being handled by the target platforms way to manage daemons (systemd, upstart, good ol' SysV init scripts). These have usually more knowledge: systemd for example will happily accept a daemon which does not fork at all, allowing it to monitor its status directly and without the requirement for a PID file. You could then provide configuration files for the most common solutions (currently probably systemd, given that Debian migrates to it too and it will thus also hit ubuntu soon), which are usually easier to write than a full fledged daemon process management.