Python / Pexpect before output out of sync

Question

I'm using Python/Pexpect to spawn an SSH session to multiple routers. The code will work for one router but then the output of session.before will get out of sync with some routers so that it will return the output from a previous sendline. This seems particularly the case when sending a blank line (sendline()). Anyone got any ideas? Any insight would be really appreciated.

Below is a sample of what I'm seeing:

ssh_session.sendline('sh version')
while (iresult==2):
    iresult = ssh_session.expect(['>','#','--More--'],timeout=SESSION_TIMEOUT)
    debug_print("execute_1 " + str(iresult))
    debug_print("execute_bef " + ssh_session.before)
    debug_print("execute_af " + ssh_session.after)

    thisoutput = ssh_session.before
    output += thisoutput

    if(iresult==2):
        debug_print("exec MORE")
        ssh_session.send(" ")
    else:
        debug_print("exec: end loop")

for cmd in config_commands:
    debug_print("------------------------------------------------\n")
    debug_print ("running command " + cmd.strip() + "\n")
    iresult=2
    ssh_session.sendline(cmd.strip())
    while (iresult==2):
        iresult = ssh_session.expect([prompt+">",prompt+"#"," --More-- "],timeout=SESSION_TIMEOUT)
        thisoutput = ssh_session.before
        debug_print("execute_1 " + str(iresult))
        debug_print("execute_af " + ssh_session.after)
        debug_print("execute_bef " + thisoutput)
        thisoutput = ssh_session.before
        output += thisoutput

        if(iresult==2):
           debug_print("exec MORE")
           ssh_session.send(" ")
        else:
           debug_print("exec: end loop")


I get this:

logged in
exec: sh version
execute_1 1
execute_bef 
R9
execute_af #
exec: end loop
------------------------------------------------

running command config t

execute_1 1
execute_af #
execute_bef sh version
Cisco IOS Software, 1841 Software (C1841-IPBASEK9-M), Version 15.1(4)M4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport...

Answer 1

I've run into this before with pexpect (and I'm trying to remember how I worked around it).

You can re-synchronize with the terminal session by sending a return and then expecting for the prompt in a loop. When the expect times out then you know that you are synchronized.

The root cause is probably that you are either:

• Calling send without a match expect (because you don't care about the output)

• Running a command that produces output but expecting for a pattern in the middle of that output and then not to next prompt that is at end of the output. One way to deal with this is to change your expect pattern to "(.+)PROMPT" - this will expect until the next prompt and capture all the output of the command sent (which you can parse in the next step).

Answer 2

I faced a similar problem. I tried waiting for the command to be printed on the screen and the sending enter.

I you want to execute say command 'cmd', then you do:

    session.send(cmd)
    index = session.expect([cmd, pexpect.TIMEOUT], 1)
    session.send('\n')
    index = session.expect([whatever you expect])

Worked for me.

Answer 3

I'm not sure this is the root of your problem, but it may be worth a try.

Something I've run into is that when you spawn a session that starts with or lands you in a shell, you have to deal with quirks of the TERM type (vt220, color-xterm, etc.). You will see characters used to move the cursor or change colors. The problem is almost guaranteed to show up with the prompt; the string you are looking for to identify the prompt appears twice because of how color changes are handled (the prompt is sent, then codes to backspace, change the color, then the prompt is sent again... but expect sees both instances of the prompt).

Here's something that handles this, guaranteed to be ugly, hacky, not very Pythonic, and functional:

import pexpect

# wait_for_prompt: handle terminal prompt craziness
#   returns either the pexpect.before contents that occurred before the 
#   first sighting of the prompt, or returns False if we had a timeout
#
def wait_for_prompt(session, wait_for_this, wait_timeout=30):
    status = session.expect([wait_for_this, pexpect.TIMEOUT, pexpect.EOF], timeout=wait_timeout)
    if status != 0:
        print 'ERROR : timeout waiting for "' + wait_for_this + '"'
        return False
    before = session.before # this is what we will want to return
    # now look for and handle any additional sightings of the prompt
    while True:
        try:
            session.expect(wait_for_this, timeout=0.1)
        except:
            # we expect a timeout here. All is normal. Move along, Citizen.
            break # get out of the while loop
        return before

s = pexpect.spawn('ssh me@myserver.local')
s.expect('password') # yes, we assume that the SSH key is already there
                     # and that we will successfully connect. I'm bad.
s.sendline('mypasswordisverysecure') # Also assuming the right password
prompt = 'me$'
wait_for_prompt(s, prompt)
s.sendline('df -h') # how full are my disks?
results = wait_for_prompt(s, prompt)
if results:
    print results
    sys.exit(0)
else:
    print 'Misery. You lose.'
    sys.exit(1)

Answer 4

I know this is an old thread, but I didn't find much about this online and I just got through making my own quick-and-dirty workaround for this. I'm also using pexpect to run through a list of network devices and record statistics and so forth, and my pexpect.spawn.before will also get out of sync sometimes. This happens very often on the faster, more modern devices for some reason.

My solution was to write an empty carriage return between each command, and check the len() of the .before variable. If it's too small, it means it only captured the prompt, which means it must be at least one command behind the actual ssh session. If that's the case, the program sends another empty line to move the actual data that I want into the .before variable:

def new_line(this, iteration):
    if iteration > 4:
        return data
    else:
        iteration+=1
        this.expect(":")
        this.sendline(" \r")
        data = this.before
        if len(data) < 50:
        # The numer 50 was chosen because it should be longer than just the hostname and prompt of the device, but shorter than any actual output
            data = new_line(this, iteration)
        return data

def login(hostname):
    this = pexpect.spawn("ssh %s" % hostname)
    stop = this.expect([pexpect.TIMEOUT,pexpect.EOF,":"], timeout=20)
    if stop == 2:
        try:
            this.sendline("\r")
            this.expect(":")
            this.sendline("show version\r")
            version = new_line(this,0)
            this.expect(":")
            this.sendline("quit\r")
            return version
        except:
            print 'failed to execute commands'
            this.kill(0)
    else:
        print 'failed to login'
        this.kill(0)

I accomplish this by a recursive command that will call itself until the .before variable finally captures the command's output, or until it calls itself 5 times, in which case it simply gives up.