Tools for implementing a watchdog timer in python

Question

I'm writing some code for testing multithreaded programs (student homework--likely buggy), and want to be able to detect when they deadlock. When running properly, the programs regularly produce output to stdout, so that makes it fairly straightforward: if no output for X seconds, kill it and report deadlock. Here's the function prototype:

def run_with_watchdog(command, timeout):
    """Run shell command, watching for output.  If the program doesn't
     produce any output for <timeout> seconds, kill it and return 1.  
     If the program ends successfully, return 0."""

I can write it myself, but it's a bit tricky to get right, so I would prefer to use existing code if possible. Anyone written something similar?

---

Ok, see solution below. The subprocess module might also be relevant if you're doing something similar.

Answer 1

You can use expect (tcl) or pexpect (python) to do this.

import pexpect
c=pexpect.spawn('your_command')
c.expect("expected_output_regular_expression", timeout=10)

Answer 2

Here's a very slightly tested, but seemingly working, solution:

import sys
import time
import pexpect
# From http://pypi.python.org/pypi/pexpect/

DEADLOCK = 1

def run_with_watchdog(shell_command, timeout):
    """Run <shell_command>, watching for output, and echoing it to stdout.
    If the program doesn't produce any output for <timeout> seconds,
    kill it and return 1.  If the program ends successfully, return 0.
    Note: Assumes timeout is >> 1 second. """

    child = pexpect.spawn('/bin/bash', ["-c", shell_command])
    child.logfile_read = sys.stdout
    while True:
        try:
            child.read_nonblocking(1000, timeout)
        except pexpect.TIMEOUT:
            # Child seems deadlocked.  Kill it, return 1.
            child.close(True)
            return DEADLOCK
        except pexpect.EOF:
            # Reached EOF, means child finished properly.
            return 0
        # Don't spin continuously.
        time.sleep(1)

if __name__ == "__main__":
    print "Running with timer..."
    ret = run_with_watchdog("./test-program < trace3.txt", 10) 
    if ret == DEADLOCK:
        print "DEADLOCK!"
    else:
        print "Finished normally"

Answer 3

Another solution:

class Watchdog:
  def __init__(self, timeout, userHandler=None): # timeout in seconds
    self.timeout = timeout
    if userHandler != None:
      self.timer = Timer(self.timeout, userHandler)
    else:
      self.timer = Timer(self.timeout, self.handler)

  def reset(self):
    self.timer.cancel()
    self.timer = Timer(self.timeout, self.handler)

  def stop(self):
    self.timer.cancel()

  def handler(self):
    raise self;

Usage if you want to make sure function finishes in less than x seconds:

watchdog = Watchdog(x)
try
  ... do something that might hang ...
except Watchdog:
  ... handle watchdog error ...
watchdog.stop()

Usage if you regularly execute something and want to make sure it is executed at least every y seconds:

def myHandler():
  print "Watchdog expired"

watchdog = Watchdog(y, myHandler)

def doSomethingRegularly():
  ...
  watchdog.reset()