[英]Redirecting stdout with subprocess in python is very slow
When I use subprocess in Python to redirect stdout, I get a very slow throughput. 当我在Python中使用子进程重定向标准输出时,吞吐量会非常缓慢。 Am I doing it wrong? 我做错了吗?
Basically, I pipe the standard output of an external program to put it in a queue. 基本上,我通过管道将外部程序的标准输出放入队列中。 Then in another function, I print it in the console. 然后在另一个功能中,我在控制台中打印它。
Here is a sample code with hexdump to generate random output: 这是带有hexdump的示例代码,用于生成随机输出:
from subprocess import Popen, PIPE
from queue import Queue
import sys
from threading import Thread, Event
import threading
class Buffer(Queue):
def __init__(self, *args, **kwargs):
Queue.__init__(self, *args, **kwargs)
def write(self, line):
self.put_nowait(line)
self.join()
def read(self):
element = self.get_nowait()
self.task_done()
return element
def write_output(buffer, stopped):
hexdump = Popen(['hexdump', '-C', '/dev/urandom'], stdout=PIPE)
while hexdump.returncode is None:
for line in hexdump.stdout.readlines(8192):
buffer.write(line)
if stopped.is_set():
hexdump.terminate()
hexdump.wait()
print('process terminated.')
break
def read_output(buffer, stopped):
while not stopped.is_set():
while not buffer.empty():
output = buffer.read()
print('********* output: {}'.format(output))
sys.stdout.flush()
print('stopped')
sys.stdout.flush()
buffer = Buffer()
stopped = Event()
generate_random_output = Thread(target=write_output, args=(buffer, stopped))
generate_random_output.name = 'generate_random_output'
generate_random_output.start()
process_output = Thread(target=read_output, args=(buffer, stopped))
process_output.name = 'process_output'
process_output.start()
try:
while True:
continue
except KeyboardInterrupt:
stopped.set()
generate_random_output.join()
process_output.join()
print('finished generating')
print('finished processing')
I would appreciate any help. 我将不胜感激任何帮助。
Instead of redirecting your output to Queue - process it directly: 无需将输出重定向到Queue,而是直接处理它:
def write_output(buffer, stopped):
hexdump = Popen(['hexdump', '-C', '/dev/urandom'], stdout=PIPE)
while hexdump.poll() is None:
while not stopped.is_set():
for line in iter(hexdump.stdout.readline, b''):
print('********* output: %s' % line.decode(), end='')
sys.stdout.flush()
hexdump.terminate()
hexdump.wait()
print('process terminated.')
break
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.