[英]real time logging to file with python subprocess
I expect this is really simple but I can't work this out. 我希望这真的很简单,但我无法解决。
I am trying to write to a log file in real time the output from a DD
imaging subprocess - I'm using DD v 8.25
from which you can get regular progress updates using the 'status=progress'
option which writes to stderr
. 我正在尝试将DD
映像子进程的输出实时写入日志文件-我使用的是DD v 8.25
,您可以使用写入到stderr
的'status=progress'
选项从中获得定期的进度更新。
I can get it to log the full output real time by passing the file object to the stderr
ie 我可以通过将文件对象传递给stderr
来获取实时记录完整输出的信息,即
log_file = open('mylog.log', 'a')
p = subprocess.Popen['dd command...'], stdout=None, stderr=log_file)
...but I would prefer to intercept the string from stderr
first so I can parse it before writing to file. ...但是我希望先从stderr
截取字符串,以便在写入文件之前可以对其进行解析。
I have tried threading but I can't seem to get it to write, or if it does, it only does it at the end of the process and not during. 我已经尝试过线程化,但是似乎无法编写它,或者如果这样做,它只会在过程结束时才执行,而不是在执行过程中。
I'm a python noob so example code would be appreciated. 我是python noob,因此示例代码将不胜感激。 Thanks! 谢谢!
UPDATE - NOW WORKING (ISH) 更新-现在工作(ISH)
I had a look at the link JF Sebastian suggested and found posts about using threads, so after that I used the "kill -USR1" trick to get DD to post progress to stderr which I could then pick up: 我看了JF Sebastian建议的链接,并找到了有关使用线程的帖子,因此在那之后,我使用了“ kill -USR1”技巧来使DD将进度发布到stderr,然后可以将其提取:
#! /usr/bin/env python
from subprocess import PIPE, Popen
from threading import Thread
from queue import Queue, Empty
import time
q = Queue()
def parsestring(mystr):
newstring = mystr[0:mystr.find('bytes')]
return newstring
def enqueue(out, q):
for line in proc1.stderr:
q.put(line)
out.close()
def getstatus():
while proc1.poll() == None:
proc2 = Popen(["kill -USR1 $(pgrep ^dd)"], bufsize=1, shell=True)
time.sleep(2)
with open("log_file.log", mode="a") as log_fh:
start_time = time.time()
#start the imaging
proc1 = Popen(["dd if=/dev/sda1 of=image.dd bs=524288 count=3000"], bufsize=1, stderr=PIPE, shell=True)
#define and start the queue function thread
t = Thread(target=enqueue, args=(proc1.stderr, q))
t.daemon = True
t.start()
#define and start the getstatus function thread
t_getstatus = Thread(target=getstatus, args=())
t_getstatus.daemon
t_getstatus.start()
#get the string from the queue
while proc1.poll() == None:
try: nline = q.get_nowait()
except Empty:
continue
else:
mystr = nline.decode('utf-8')
if mystr.find('bytes') > 0:
log_fh.write(str(time.time()) + ' - ' + parsestring(mystr))
log_fh.flush()
#put in a delay
#time.sleep(2)
#print duration
end_time=time.time()
duration=end_time-start_time
print('Took ' + str(duration) + ' seconds')
The only issue is I can't work out how to improve performance. 唯一的问题是我不知道如何提高性能。 I only need it to report status every 2 seconds or so but increasing the time delay increases the time of the imaging, which I don't want. 我只需要每2秒钟左右报告一次状态,但是增加时间延迟会增加成像时间,这是我不希望的。 That's a question for another post though... 这是另一个帖子的问题...
Thanks to both JF Sebastian and Ali. 感谢JF Sebastian和Ali。
With this example it's possible (with python 3) to stream from stderr to console: 在此示例中,可以(使用python 3)从stderr流式传输到控制台:
#! /usr/bin/env python
from subprocess import Popen, PIPE
# emulate a program that write on stderr
proc = Popen(["/usr/bin/yes 1>&2 "], bufsize=512, stdout=PIPE, stderr=PIPE, shell=True)
r = b""
for line in proc.stderr:
r += line
print("current line", line, flush=True)
To stream to a file: 要流式传输到文件:
#! /usr/bin/env python
from subprocess import Popen, PIPE
with open("log_file.log", mode="b", encoding="utf8") as log_fh:
proc = Popen(["/usr/bin/yes 1>&2 "], bufsize=512, stdout=PIPE, stderr=PIPE, shell=True)
r = b""
# proc.stderr is an io.TextIOWrapper file-like obj
# iter over line
for line in proc.stderr:
r += line
# print("current line", line, flush=True)
log_fh.write(line) # file open in binary mode
# log_fh.write(line.decode("utf8")) # for text mode
log_fh.flush() # flush the content
To display dd
's progress report in a terminal and to save (parsed) output to a log file: 要在终端中显示dd
的进度报告,并将输出(解析的)保存到日志文件中:
#!/usr/bin/env python3
import io
from subprocess import PIPE, Popen
from time import monotonic as timer
cmd = "dd if=/dev/sda1 of=image.dd bs=524288 count=3000 status=progress".split()
with Popen(cmd, stderr=PIPE) as process, \
open("log_file.log", "a") as log_file:
start_time = timer()
for line in io.TextIOWrapper(process.stderr, newline=''):
print(line, flush=True, end='') # no newline ('\n')
if 'bytes' in line:
# XXX parse line here, add flush=True if necessary
print(line, file=log_file)
# print duration
print('Took {duration} seconds'.format(duration=timer() - start_time))
shell=True
: you don't need the shell here. no shell=True
:这里不需要shell。 Popen()
can run dd
directly Popen()
可以直接运行dd
while proc1.poll() == None
You don't need it here (you'll see EOF on proc1.stderr
if proc1.poll()
is not None). 请,请不要while proc1.poll() == None
时使用此处您不需要它(如果proc1.poll()
不为None,您将在proc1.stderr
上看到EOF)。 You may lose data (there could be a buffered content even if the process has exited already). 您可能会丢失数据(即使进程已经退出,也可能存在缓冲内容)。 Unrelated: if you need to compare with None
; 无关:如果需要与None
比较; use is None
instead of == None
use is None
而不是== None
io.TextIOWrapper(newline='')
enables text mode (it uses locale.getpreferredencoding(False)
) and it treats '\\r'
as a newline too io.TextIOWrapper(newline='')
启用文本模式(它使用locale.getpreferredencoding(False)
),并且也将'\\r'
视为换行符 bufsize=-1
(see io.DEFAULT_BUFFER_SIZE
) 使用默认的bufsize=-1
(请参阅io.DEFAULT_BUFFER_SIZE
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.