從gevent-subprocess獲取實時標准輸出？

Question

我試圖通過POPEN盡快得到一個流程的stdout。 使用gevent 1.0 readline（）和read（）仍然阻止進程並等待進程完成。 有線索嗎？ 是的，我搜索了一個簡單的解決方案。 它必須是沒有線程的，對吧？

Answer 1

import gevent
from gevent.subprocess import Popen, PIPE

def cron():
    while True:
        print("cron")
        gevent.sleep(0.5)

g = gevent.spawn(cron)
def subp():
    sub = Popen('sleep 1; ping www.google.com -c 2; sleep 5; uname', stdout=PIPE, shell=True)
    while True:
        s = sub.stdout.readline()
        if s == "":
            break
        else:
            print s.strip()
    g.kill()
subp()

也可以檢查這個名為prun的工具的要點： https ： //gist.github.com/zhangchunlin/05576572b628f5bf9d74

Answer 2

我用一點點黑客解決了這個問題。 只要將一行寫入stdout，只需刷新Pythons行緩沖區w / self.stream.flush（）和一個輔助類。

Answer 3

這里是一個可以用來讀取標准輸出和標准錯誤並發的技術。 它假設您可以提供一個緩沖區來寫入從流中讀取的數據，但正如您所看到的，它可以通過調用日志記錄模塊，簡單的打印語句或回調來輕松替換，或者在運行中執行某些操作，或者：

import gevent
import gevent.subprocess

def read_stream(stream, buf):

    try:
        while not stream.closed:
            l = stream.readline()
            if not l: break
            buf.write(l)
    except RuntimeError:
        # process was terminated abruptly
        pass


p = gevent.subprocess.Popen(...)
stdout = ... #create buffer with write() method
stderr = ... #create buffer with write() method

gevent.spawn(read_stream, p.stdout, stdout)
gevent.spawn(read_stream, p.stderr, stderr)

status = p.wait()

編輯 ：在JF Sebastien的評論之后，我實現了一個完整的示例，演示了使用greenlet的輸出的並發，實時讀數。 我沒有累積上面提出的輸出，但只是為了這個例子打印它。 這里是：

import gevent
import gevent.subprocess

def count_greenlets(s):
    '''See: http://stackoverflow.com/a/20027162/712525'''

    import gc
    from greenlet import greenlet
    greenlets = [obj for obj in gc.get_objects() if isinstance(obj, greenlet)]
    print('At "%s", greenlets: %d' % (s, len(greenlets)))
    for k in greenlets:
        print('  * %s' % (k,))


def read_stream(stream):

    try:
        while not stream.closed:
            l = stream.readline()
            if not l: break
            print(l.rstrip())
    except RuntimeError:
        # process was terminated abruptly
        pass


count_greenlets('start')

p1 = gevent.subprocess.Popen('ping -c 5 www.google.com', stdout=gevent.subprocess.PIPE, stderr=gevent.subprocess.PIPE, shell=True)
gevent.spawn(read_stream, p1.stdout)
gevent.spawn(read_stream, p1.stderr)

count_greenlets('after p1')

p2 = gevent.subprocess.Popen('ping -c 5 www.facebook.com', stdout=gevent.subprocess.PIPE, stderr=gevent.subprocess.PIPE, shell=True)
gevent.spawn(read_stream, p2.stderr)
gevent.spawn(read_stream, p2.stdout)

count_greenlets('after p2')

p1.wait()
count_greenlets('after p1 wait')
p2.wait()
count_greenlets('after p2 wait')

count_greenlets('end')

它給出了以下輸出：

At "start", greenlets: 1
  * <greenlet.greenlet object at 0x1060d0690>
At "after p1", greenlets: 4
  * <Hub at 0x106300f50 select default pending=0>
  * <Greenlet at 0x10646b0f0: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <Greenlet at 0x10646b190: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <greenlet.greenlet object at 0x1060d0690>
At "after p2", greenlets: 6
  * <Hub at 0x106300f50 select default pending=0>
  * <Greenlet at 0x10646b0f0: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <Greenlet at 0x10646b190: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <Greenlet at 0x10646b230: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <Greenlet at 0x10646b2d0: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <greenlet.greenlet object at 0x1060d0690>
PING www.google.com (172.217.19.164): 56 data bytes
64 bytes from 172.217.19.164: icmp_seq=0 ttl=56 time=12.722 ms
PING star-mini.c10r.facebook.com (31.13.91.36): 56 data bytes
64 bytes from 31.13.91.36: icmp_seq=0 ttl=87 time=29.673 ms
64 bytes from 172.217.19.164: icmp_seq=1 ttl=56 time=11.863 ms
64 bytes from 31.13.91.36: icmp_seq=1 ttl=87 time=31.389 ms
64 bytes from 172.217.19.164: icmp_seq=2 ttl=56 time=13.492 ms
64 bytes from 31.13.91.36: icmp_seq=2 ttl=87 time=29.921 ms
64 bytes from 172.217.19.164: icmp_seq=3 ttl=56 time=12.488 ms
64 bytes from 31.13.91.36: icmp_seq=3 ttl=87 time=30.859 ms
64 bytes from 172.217.19.164: icmp_seq=4 ttl=56 time=13.053 ms

--- www.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.863/12.724/13.492/0.547 ms
At "after p1 wait", greenlets: 4
  * <Hub at 0x106300f50 select default pending=0>
  * <Greenlet at 0x10646b230: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <Greenlet at 0x10646b2d0: read_stream(<gevent._fileobjectposix.FileObjectPosix object at)>
  * <greenlet.greenlet object at 0x1060d0690>
64 bytes from 31.13.91.36: icmp_seq=4 ttl=87 time=30.379 ms

--- star-mini.c10r.facebook.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 29.673/30.444/31.389/0.622 ms
At "after p2 wait", greenlets: 2
  * <Hub at 0x106300f50 select default pending=0>
  * <greenlet.greenlet object at 0x1060d0690>
At "end", greenlets: 2
  * <Hub at 0x106300f50 select default pending=0>
  * <greenlet.greenlet object at 0x1060d0690>