pexpect sendline is too slow

Question

I'm running a subprocess using pexpect. I'm communicating with it off and on via sendline and readline. cProfile indicates my program spends most of its time waiting around for sendline to finish, which is not desired behavior. Python pseudocode:

def reset(self):
  self.proc = pexpect.spawn("node ./my_lovely_program.js")
...
def step(self, info):
    self.proc.sendline(info)
    while (my_condition):
        response = self.proc.readline().decode()
        # evaluate my_condition
    return retval

Meanwhile, in the style of Waiting for user to enter input in Node.js , buried well into my node application is:

let ans = await this.askQuestion("");
// do_stuff(ans);

This gets run by two different instances of the relevant class at the same time. This is all well and good and runs satisfactorily. Then I run

python3 -m cProfile -s cumtime main.py

And get this:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    498/1    0.002    0.000  262.820  262.820 {built-in method builtins.exec}
        1    0.107    0.107  262.820  262.820 main loop
        1    0.099    0.099  262.081  262.081 also main loop
     3774    0.022    0.000  214.272    0.057 main.py:24(step)
    33596  203.977    0.006  203.977    0.006 {built-in method time.sleep}
     3774    0.017    0.000  189.138    0.050 pty_spawn.py:570(sendline)
     3774    0.049    0.000  189.105    0.050 pty_spawn.py:526(send)
     3875    0.158    0.000   53.767    0.014 main.py:28(scrape_input)
    14756    0.052    0.000   53.434    0.004 spawnbase.py:459(readline)
    14756    0.056    0.000   53.382    0.004 spawnbase.py:240(expect)
    14756    0.072    0.000   52.985    0.004 spawnbase.py:343(expect_list)
    14756    0.242    0.000   52.848    0.004 expect.py:91(expect_loop)
    29721    0.235    0.000   47.113    0.002 pty_spawn.py:415(read_nonblocking)
    69657    0.113    0.000   46.250    0.001 pty_spawn.py:448(select)
    69657    0.103    0.000   46.137    0.001 utils.py:130(select_ignore_interrupts)
    69657   46.016    0.001   46.016    0.001 {built-in method select.select}
...

Between them, readline and especially sendline take up over 90% of the programs runtime. Most of that time is spent sleeping. This is very sad and I want to fix it. I'm sending 20-30 characters each time. I've tried reading from stdin in other ways (eg https://www.dev2qa.com/node-js-get-user-input-from-command-line-prompt-example/ ) and am ready to keep going down that route if that's the way to go. (The issue I am having there is a separate one; on the second communication, the input gets read twice). Posix_spawn performance when capturing process output used popen; I remember using that several months ago and it deadlocking.

Answer 1

pexpect has a default 50ms wait time for send(). You can disable this by setting:

proc.delaybeforesend = None

See the documentation for an explanation of why they put this delay in.