如何从“子进程”流式输出（一次一行）？

Question

我真的很难使用subprocess包在我的 Python 脚本中成功地从 bash 命令流输出。 该命令只是一个简单的AWS CLI命令，用于将对象上传到 S3：

aws s3 cp --profile MY-PROFILE UPLOADED_FILE s3://BUCKET/PREFIX/

当我在bash运行它时，它会显示上传的进度。 使用subprocess ，我只能通过以下方式使其工作：

• 最终输出仅在最后。

   upload: ./OBJECT to s3://BUCKET/PREFIX/OBJECT

• 逐行输出

  Completed 512.0 KiB/12.2 MiB (697.8 KiB/s) with 1 file(s) remaining Completed 768.0 KiB/12.2 MiB (980.0 KiB/s) with 1 file(s) remaining Completed 1.0 MiB/12.2 MiB (1.2 MiB/s) with 1 file(s) remaining Completed 1.2 MiB/12.2 MiB (1.5 MiB/s) with 1 file(s) remaining Completed 1.5 MiB/12.2 MiB (1.6 MiB/s) with 1 file(s) remaining Completed 1.8 MiB/12.2 MiB (1.4 MiB/s) with 1 file(s) remaining

• 类似的东西没有线分离：

   b'Completed 256.0 KiB/12.2 MiB (297.5 KiB/s) with 1 file(s) remaining\\rCompleted 512.0 KiB/12.2 MiB (506.7 KiB/s) with 1 file(s) remaining\\rCompleted 768.0 KiB/12.2 MiB (544.4 KiB/s) with 1 file(s) remaining\\rCompleted 1.0 MiB/12.2 MiB (672.0 KiB/s) with 1 file(s) remaining \\rCompleted 1.2 MiB/12.2 MiB (707.0 KiB/s) with 1 file(s) remaining \\rCompleted 1.5 MiB/12.2 MiB (780.3 KiB/s) with 1 file(s) remaining \\rCompleted 1.8 MiB/12.2 MiB (849.1 KiB/s) with 1 file(s) remaining \\rCompleted 2.0 MiB/12.2 MiB (865.6 KiB/s) with 1 file(s) remaining \\rCompleted 2.2 MiB/12.2 MiB (971.0 KiB/s) with 1 file(s) remaining \\rCompleted 2.5 MiB/12.2 MiB (992.4 KiB/s) with 1 file(s) remaining \\rCompleted 2.8 MiB/12.2 MiB (1.0 MiB/s) with 1 file(s) remaining

我需要的是只使用一行并显示实时状态，就像在 bash 中一样。

以下是我尝试过的许多不同功能，但都没有运气：

from subprocess import Popen, PIPE
import subprocess
import os
import shlex
from time import sleep


def stream_process(process):
    go = process.poll() is None
    for line in process.stdout:
        print(line)
    return go

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while stream_process(process):
    sleep(0.1)


def run(command):
  with subprocess.Popen(command, text=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as process:
    for line in process.stdout:
      print(line)

run(command)

process = Popen(command, text=True, shell=True, stdout=PIPE, bufsize=1, universal_newlines=True)
while True:
    output = process.stdout.readline()
    if process.poll() is not None and output == b'':
        break
    if output:
        print (output.strip())
retval = process.poll()


with Popen(command, text=True, shell=True, stdout=PIPE, bufsize=1, universal_newlines=True) as p:

    while True: 
        line = p.stdout.readline()
        print(line.strip())


    for line in p.stdout:
        print(line.strip(), "\n", flush=True, end='')
        # print(line.strip())


print(result.stdout.readlines())

print("args", result.args)
print("returncode", result.returncode)
print("stdout", result.stdout)
print("stderr", result.stderr)
print("check_returncode()", result.check_returncode())

def run_command(cmd):
    p = Popen(shlex.split(cmd), bufsize=1, universal_newlines=True)
    return p.poll()

run_command(command)
# invoke process
process = subprocess.Popen(shlex.split(command),shell=False,stdout=subprocess.PIPE)

# Poll process.stdout to show stdout live
while True:
  output = process.stdout.readline()
  if process.poll() is not None:
    break
  if output:
    print(output.strip())
rc = process.poll()


print(result.stdout.readlines())
while True:
    line = result.stdout.readline()
    if not line:
        break
    print(line, flush=True)

常问问题

你为什么不使用boto3 ，而不是subprocess + bash ？

我正在为非 Python 开发人员开发程序，因此我想限制依赖项，并尽可能只使用标准包。

Answer 1

您只需：

import subprocess

p = subprocess.Popen(("ls"), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout:
    print(line.decode("utf-8").rstrip())
p.wait()
status = p.poll()
print("process terminate with code: %s" % status)

造成：

my-file.txt
...
process terminate with code: 0

当子进程退出时，stdout 关闭，从而中断循环。

子进程标准输出是一个字节流。 如果你需要 str 对象，你应该解码它： line.decode("utf-8") 。

但我认为这不是问题。 在 stdout 是终端或管道的情况下，某些命令具有不同的输出。 就像wget一样。 我想要一个进度条，你必须解析输出并实现你自己的进度条。

Answer 2

for line in process.stdout和process.stdout.readline()不适合该任务，因为它们在整行结束之前不会返回，即收到\\n ，而进度部分仅以\\r结尾。 独立于任何行终止读取输出的一种方法是使用os.read() ：

from os         import read

process = Popen(shlex.split(cmd), stdout=PIPE)
while buffer := read(process.stdout.fileno(), 4096):
    print(buffer.decode(), end='')

正如 xrisk 指出的那样，使用text=True将允许使用readline()读取以\\r结尾的行，但这在这里没有用，因为输出中的所有行结尾都将转换为'\\n' 。

但是确实有一种方法可以通过io.TextIOWrapper在不丢失原始行结尾的情况下使用文本流 - 我们可以这样做：

import io

process = Popen(shlex.split(cmd), stdout=PIPE)
for line in io.TextIOWrapper(process.stdout, newline=""): print(line, end="")