Pexpect spawn.expect() seems to be unreliable in detecting process output

Question

I have a class, ServerManager , which monitors and interfaces with another process using pexpect . Unfortunately, there is not a cleaner way to do this. The process in question does not provide an API.

The ServerManager needs to monitor the process's output and trigger events when it recognizes specific patterns. Because there are multiple such patterns to monitor, and pexpect 's spawn.expect() blocks the current thread, these "listeners" get spun off into separate threads that interact with the main thread when they match on their pattern.

One such example is waiting for users to connect/disconnect:

import pexpect
from threading import Thread,Lock

usersLock = Lock()

class ListenerThread(Thread):

  def __init__(self, target, name=None, args=[], kwargs={}):
    super(ListenerThread, self).__init__(name=name)
    self.target = lambda: target(*args, **kwargs)
    self.isStopped = False # add a way to safely halt this thread

  def stop(self):
    self.isStopped = True

  def run(self):
    while not self.isStopped: # run until told otherwise
      try:
        self.target()
      except pexpect.TIMEOUT:
        # we can't wait forever...
        continue
      except pexpect.EOF:
        self.isStopped = True

class ServerManager(object):

  def __init__(self):
    self.process = pexpect.spawn(...) # Spawn the process
    self.numberOfUsers = 0
    # start up the listeners
    self.listeners = []
    connectListener = ListenerThread(self.waitForConnect, name="Connect listener")
    connectListener.start()
    disconnectListener = ListenerThread(self.waitForDisconnect, name="Disconnect listener")
    disconnectListener.start()
    self.listeners += [connectListener,disconnectListener] # keep track of the threads

  def waitForConnect(self):
    self.process.expect(...) # watch for the line that is printed when a user connects
    usersLock.acquire()
    try:
      self.numberOfUsers += 1
    finally:
      usersLock.release()

  def waitForDisconnect(self):
    self.serverProcess.expect(...) # watch for the line that is printed when a user disconnects
    usersLock.acquire()
    try:
      self.numberOfUsers -= 1
    finally:
      usersLock.release()

The problem is that the "connect" and "disconnect" events are triggered very unreliably. I created an instance of the ServerManager and connected/disconnected 10 times (waiting about 10 seconds between each action), checking numberOfUsers after each connect/disconnect. It was only being updated about 1/8 of the time at best, over multiple trials.

Is this an issue with thread safety in pexpect ? Is there a better way to watch for these sorts of events given that my only means of interfacing with the process is by monitoring its command-line output?

Answer 1

You have here two threads making a blocking call on the same file descriptor. I would implement this is a single threaded asynchronous event loop. The expect method should be able to watch for more than one string can invoke a callback function for each outcome from one call. I'm not sure pexpect can actually do this (I don't use it), but take a close look at it's documentation.