[英]How do I stream output (one line at a time) from `subprocess`?
我真的很難使用subprocess
包在我的 Python 腳本中成功地從 bash 命令流輸出。 該命令只是一個簡單的AWS CLI
命令,用於將對象上傳到 S3:
aws s3 cp --profile MY-PROFILE UPLOADED_FILE s3://BUCKET/PREFIX/
當我在bash
運行它時,它會顯示上傳的進度。 使用subprocess
,我只能通過以下方式使其工作:
最終輸出僅在最后。
upload: ./OBJECT to s3://BUCKET/PREFIX/OBJECT
逐行輸出
Completed 512.0 KiB/12.2 MiB (697.8 KiB/s) with 1 file(s) remaining Completed 768.0 KiB/12.2 MiB (980.0 KiB/s) with 1 file(s) remaining Completed 1.0 MiB/12.2 MiB (1.2 MiB/s) with 1 file(s) remaining Completed 1.2 MiB/12.2 MiB (1.5 MiB/s) with 1 file(s) remaining Completed 1.5 MiB/12.2 MiB (1.6 MiB/s) with 1 file(s) remaining Completed 1.8 MiB/12.2 MiB (1.4 MiB/s) with 1 file(s) remaining
類似的東西沒有線分離:
b'Completed 256.0 KiB/12.2 MiB (297.5 KiB/s) with 1 file(s) remaining\\rCompleted 512.0 KiB/12.2 MiB (506.7 KiB/s) with 1 file(s) remaining\\rCompleted 768.0 KiB/12.2 MiB (544.4 KiB/s) with 1 file(s) remaining\\rCompleted 1.0 MiB/12.2 MiB (672.0 KiB/s) with 1 file(s) remaining \\rCompleted 1.2 MiB/12.2 MiB (707.0 KiB/s) with 1 file(s) remaining \\rCompleted 1.5 MiB/12.2 MiB (780.3 KiB/s) with 1 file(s) remaining \\rCompleted 1.8 MiB/12.2 MiB (849.1 KiB/s) with 1 file(s) remaining \\rCompleted 2.0 MiB/12.2 MiB (865.6 KiB/s) with 1 file(s) remaining \\rCompleted 2.2 MiB/12.2 MiB (971.0 KiB/s) with 1 file(s) remaining \\rCompleted 2.5 MiB/12.2 MiB (992.4 KiB/s) with 1 file(s) remaining \\rCompleted 2.8 MiB/12.2 MiB (1.0 MiB/s) with 1 file(s) remaining
我需要的是只使用一行並顯示實時狀態,就像在 bash 中一樣。
以下是我嘗試過的許多不同功能,但都沒有運氣:
from subprocess import Popen, PIPE
import subprocess
import os
import shlex
from time import sleep
def stream_process(process):
go = process.poll() is None
for line in process.stdout:
print(line)
return go
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while stream_process(process):
sleep(0.1)
def run(command):
with subprocess.Popen(command, text=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as process:
for line in process.stdout:
print(line)
run(command)
process = Popen(command, text=True, shell=True, stdout=PIPE, bufsize=1, universal_newlines=True)
while True:
output = process.stdout.readline()
if process.poll() is not None and output == b'':
break
if output:
print (output.strip())
retval = process.poll()
with Popen(command, text=True, shell=True, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
while True:
line = p.stdout.readline()
print(line.strip())
for line in p.stdout:
print(line.strip(), "\n", flush=True, end='')
# print(line.strip())
print(result.stdout.readlines())
print("args", result.args)
print("returncode", result.returncode)
print("stdout", result.stdout)
print("stderr", result.stderr)
print("check_returncode()", result.check_returncode())
def run_command(cmd):
p = Popen(shlex.split(cmd), bufsize=1, universal_newlines=True)
return p.poll()
run_command(command)
# invoke process
process = subprocess.Popen(shlex.split(command),shell=False,stdout=subprocess.PIPE)
# Poll process.stdout to show stdout live
while True:
output = process.stdout.readline()
if process.poll() is not None:
break
if output:
print(output.strip())
rc = process.poll()
print(result.stdout.readlines())
while True:
line = result.stdout.readline()
if not line:
break
print(line, flush=True)
常問問題
你為什么不使用
boto3
,而不是subprocess
+bash
?
我正在為非 Python 開發人員開發程序,因此我想限制依賴項,並盡可能只使用標准包。
您只需:
import subprocess
p = subprocess.Popen(("ls"), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout:
print(line.decode("utf-8").rstrip())
p.wait()
status = p.poll()
print("process terminate with code: %s" % status)
造成:
my-file.txt
...
process terminate with code: 0
當子進程退出時,stdout 關閉,從而中斷循環。
子進程標准輸出是一個字節流。 如果你需要 str 對象,你應該解碼它: line.decode("utf-8")
。
但我認為這不是問題。 在 stdout 是終端或管道的情況下,某些命令具有不同的輸出。 就像wget
一樣。 我想要一個進度條,你必須解析輸出並實現你自己的進度條。
for line in process.stdout
和process.stdout.readline()
不適合該任務,因為它們在整行結束之前不會返回,即收到\\n
,而進度部分僅以\\r
結尾。 獨立於任何行終止讀取輸出的一種方法是使用os.read() :
from os import read
process = Popen(shlex.split(cmd), stdout=PIPE)
while buffer := read(process.stdout.fileno(), 4096):
print(buffer.decode(), end='')
正如 xrisk 指出的那樣,使用text=True
將允許使用readline()
讀取以\\r
結尾的行,但這在這里沒有用,因為輸出中的所有行結尾都將轉換為'\\n'
。
但是確實有一種方法可以通過io.TextIOWrapper
在不丟失原始行結尾的情況下使用文本流 - 我們可以這樣做:
import io
process = Popen(shlex.split(cmd), stdout=PIPE)
for line in io.TextIOWrapper(process.stdout, newline=""): print(line, end="")
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.