How to monitor a python process and restart it upon abnormal termination

Question

Assume that there is a task.py ,breaking due to memory overflow.How can i monitor this and restart it?

import time
while(1):
    print('.')
    # simulate breaks
    time.sleep(2)
    exit(0)

Thanks

Answer 1

You can use a watchdog. Make your worker process update a dummyfile say every 10 secs. Have another, completely independent, process check if the last access wasn't longer that say 20 secs ago. If it was, restart your worker process.

There are all kinds of nifty OS-dependent ways to do the same, but this low-tech one always works, even trivially over a network. Since you only measure time difference between two accesses, the clocks don't even have to be synchronized.

Answer 2

Something like this should work:

while ! /path/to/task.py; do
    echo 'restarting task...'
done

If task.py exits with non-zero exit status the loop will continue and run the script again. The loop will only break when task.py exits with 0 .

If your program is errored and yield to non-zero exit at all time, this will end up being an infinite loop. So it's better to restrict the number of restart tries by a max_try value:

#!/bin/bash
max_try=100
count=1
while ! python /path/to/task.py; do
    ((count++)) # increment (Bashism)
    #count=$(expr $count + 1) # increment (portable)
    if [ $count -gt $max_try ]; then break; fi
    echo 'restarting task...'
done

Answer 3

If it actually runs out of memory it should get OOM killed . If you have another process which restarts it continuously (for example while true; do /path/to/my_script.py; done ) it should get up and running again immediately.