Interactive, non-blocking subprocess.Popen script without using communicate or pexpect

Question

A: Why does it block?

B: How may I massage this slightly so that it will run without blocking?

#!/usr/bin/env python
import subprocess as sp
import os

kwds = dict(
    stdin=sp.PIPE,
    stdout=sp.PIPE,
    stderr=sp.PIPE,
    cwd=os.path.abspath(os.getcwd()),
    shell=True,
    executable='/bin/bash',
    bufsize=1,
    universal_newlines=True,
)
cmd = '/bin/bash'
proc = sp.Popen(cmd, **kwds)
proc.stdin.write('ls -lashtr\n')
proc.stdin.flush()

# This blocks and never returns
proc.stdout.read()

I need this to run interactively.

This is a simplified example, but the reality is I have a long running process and I'd like to startup a shell script that can more or less run arbitrary code (because it's an installation script).

EDIT: I would like to effectively take a .bash_history over several different logins, clean it up so it is a single script, and then execute the newly crafted shell script line-by-line within a shell stored within a Python script.

For example:

> ... ssh to remote aws system ...
> sudo su -
> apt-get install stuff
> su - $USERNAME
> ... create and enter a docker snapshot ...
> ... install packages, update configurations
> ... install new services, update service configurations ...
> ... drop out of snapshot ...
> ... commit the snapshot ...
> ... remove the snapshot ...
> ... update services ...
> ... restart services ...
> ... drop into a tmux within the new docker ...

This takes hours manually; it should be automated.

Answer 1

A: Why does it block?

It blocks because that's what .read() does: it reads all of the bytes until an end-of-file indication. Since the process never indicates end of file, the .read() never returns.

B: How may I massage this slightly (emphasis on slightly) so that it will run without blocking?

One thing to do is to cause the process to indicate end of file. A small change is to cause the subprocess to exit.

proc.stdin.write('ls -lashtr; exit\n')

Answer 2

This is an example form my another answer: https://stackoverflow.com/a/43012138/3555925 , which did not use pexpect. You can see more detail in that answer.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import sys
import select
import termios
import tty
import pty
from subprocess import Popen

command = 'bash'
# command = 'docker run -it --rm centos /bin/bash'.split()

# save original tty setting then set it to raw mode
old_tty = termios.tcgetattr(sys.stdin)
tty.setraw(sys.stdin.fileno())

# open pseudo-terminal to interact with subprocess
master_fd, slave_fd = pty.openpty()

# use os.setsid() make it run in a new process group, or bash job control will not be enabled
p = Popen(command,
          preexec_fn=os.setsid,
          stdin=slave_fd,
          stdout=slave_fd,
          stderr=slave_fd,
          universal_newlines=True)

while p.poll() is None:
    r, w, e = select.select([sys.stdin, master_fd], [], [])
    if sys.stdin in r:
        d = os.read(sys.stdin.fileno(), 10240)
        os.write(master_fd, d)
    elif master_fd in r:
        o = os.read(master_fd, 10240)
        if o:
            os.write(sys.stdout.fileno(), o)

# restore tty settings back
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_tty)