How can I monitor a Python process and relaunch if it dies?

Question

I have a program that I'm running that listens to a queue (it's not multi-threaded so I want to run several instances of it). I've tried my best to catch errors but in the event the app crashes due to a error or bad incoming data, I want to be able to respawn the Python app (after I log the stacktrace) so it continues to work.

I felt this might be a common problem for people who run python based services so I thought I would ask but I was thinking of writing some code to do a ps -ef and count the instances of the name of the Python program (if less than a threshold then I would have the program relaunch it).

Before I build this, I wanted to know if there was perhaps a better way or a existing tool/module that did this?

Answer 1

Checkout supervisord . I use it regularly to launch, monitor all types of things.

Here is how I set it up to launch a wsgi app on my server:

[program:quizzes]
directory = /var/www/quizzes.seasources.net
command = /home/jaime/code/virtualenv/quizzes/bin/uwsgi uwsgi.ini
process_name = quizzes
autostart = true
startsecs = 5
user = www-data
redirect_stderr = true
stdout_logfile = /var/www/quizzes.seasources.net/logs/supervisor-console.log
environment = PYTHON_EGG_CACHE=/tmp/python-eggs

The configuration file format is easy to understand and it even logs stdout/stderr to a file. Above it's /var/www/quizzes.seasources.net/logs/supervisor-console.log You can read more about configuration here .

Answer 2

You could use a supervisor. A well known one that's written in Python would be supervisord , a more recent one also in Python would be Circus , and then there are Monit or daemontools and probably many more.

Answer 3

If you are looking for something more simple, you can use the subprocess module (python default) to start and check your processes... A basic version would look like this:

# run.py

import subprocess, time

# add your listener processor call here
_PROCESS_ARGS = ['python','/path/to/listener.py']
_PROCESS_TOTAL = 10

process_list = []

# start the processes...
for i in range(_PROCESS_TOTAL):     
    process_list.append(subprocess.Popen(_PROCESS_ARGS))

while True:     
    for i in range(_PROCESS_TOTAL):         
        p = process_list[i]         
            if p.poll() != None: # check if process is running                      
                process_list[i] = subprocess.Popen(_PROCESS_ARGS) # if not, replace with new one
    time.sleep(1) # check only every second...