Capture output from C program on stdout using Python subprocess? Pexpect?

Question

I want to capture output from a C program I'm launching like this:

p = subprocess.Popen(["make", "run_pci"],
                     stdout=subprocess.PIPE,
                     cwd="/home/ecorbett/hello_world_pthread")
for ln in p.stdout:

Only problem is I dont get output until the C program is done, when in fact I need to get output line by line as the program is running. And to further complicate matters, I have to parse each line ( i only need certain data from lines).

For example, here is some sample output: (I need to capture "Thread on Tile #")

blahblah blah Thread blahblah blah tile 1: On 
blahblah blah Thread blahblah blah tile 2: OFF 
blahblah blah Thread blahblah blah tile 3 : Disable

I noticed the article I linked below seems to have the same problem. I was trying to figure how to adapt it to my situation?

Getting realtime output from ffmpeg to be used in progress bar (PyQt4, stdout)

Python newbie, so example code is greatly appreciated!!!

Answer 1

The reason you need to use pexpect is that the program's stdio will use block buffering if it is not connected to a tty. pexpect uses a pseudo-tty (pty) so stdio will use line buffering and you will be able to access lines as they are output.

Change your code to:

p = pexpect.spawn('make', ['run_pci'], cwd="/home/ecorbett/hello_world_pthread")
for ln in p:
    ...

You can use pexpect.spawn.expect to just get the output you're interested in:

while p.expect('Thread on Tile (\d+):', pexpect.EOF) == 0:
    print("Tile {0}".format(int(p.group[1])))

Answer 2

You can't use p.stdout like that; if you ask for "the whole stdout", this will only be available upon process termination (or pipe buffer filling, which could take a long time).

You need to read from the process's stdout line by line.

while True:
    ln = p.stdout.readline()
    if '' == ln:
        break
    m = re.search("Thread (?P<id>\d+)", ln);
    if m:
        # use m.group() to extract information
        # e.g. m.group('id') will hold the 12345 from "Thread 12345"

It would also be best if stdout could be set to line buffered (usually it is fully buffered wherever possible), but I think this can only be done from within the called program.

We have two buffers to consider here. One is the C program's output buffer. This may be nonexistent (unbuffered output), line buffered, or fully buffered (1K, 4K or 8K are some possible sizes).

Within the program, a "printf()" is called. The output goes:

• out, if unbuffered
• into the buffer; and then all newline-terminated lines in buffer are output, if line buffered;
• into the buffer; and then the first 4K are output, if fully buffered with 4K buffer and the buffer is fuller than 4K.

Now the output enters Python's pipe. This again may be fully buffered (stdout) or line buffered (readline). So the output goes:

• to the python program's logic, if there's one full newline-terminated line in the pipeline and we're using readline
• to the buffer, if there's less than 4K in the pipeline and we're using "for ln in stdout".

In this last case, the buffer will go in 4K chunks to the Python logic.

Let us now imagine a line buffered C program outputting one line, 1K characters long, each second, to a Python program (if the C program is fully buffered, there's not very much that can be done!)

Reading stdout in cycle, we would see (inside the for loop):

• t = 0 ... nothing
• t = 1 ... nothing (buffer is 50% full)
• t = 2 ... nothing (buffer is 75% full)
• t = 3 ... FOUR lines of output
• t = 4 ... nothing ...

Reading through readline we would get:

• t = 0 ... one line
• t = 1 ... one line
• t = 2 ... one line
• t = 3 ... one line

EXAMPLE

Here I run "ping -c 3 -i 2 127.0.0.1" in order to get three packets to localhost at two seconds interval. One run of ping takes around six seconds. I read the output from ping, and print a timestamp. The whole output of ping is small enough that it fits in Python's full-buffer.

#!/usr/bin/python

import subprocess
from time import gmtime, strftime

p = subprocess.Popen(["ping", "-c", "3", "-i", "2", "127.0.0.1"],
                 stdout=subprocess.PIPE)

for ln in p.stdout:
    print strftime("%H:%M:%S", gmtime()) + " received " + ln

# Now I start the same process again, reading the input the other way.

p = subprocess.Popen(["ping", "-c", "3", "-i", "2", "127.0.0.1"],
                 stdout=subprocess.PIPE)

while True:
    ln = p.stdout.readline()
    if '' == ln:
            break
    print strftime("%H:%M:%S", gmtime()) + " received " + ln

The output I receive on my Linux box is, as expected:

(nothing for the first six seconds)
15:40:10 received PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
15:40:10 received 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.037 ms
15:40:10 received 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.034 ms
15:40:10 received 64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.031 ms
15:40:10 received
15:40:10 received --- 127.0.0.1 ping statistics ---
15:40:10 received 3 packets transmitted, 3 received, 0% packet loss, time 3998ms
15:40:10 received rtt min/avg/max/mdev = 0.031/0.034/0.037/0.002 ms

15:40:10 received PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
15:40:10 received 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.041 ms
15:40:12 received 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.039 ms
15:40:14 received 64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.035 ms
15:40:14 received
15:40:14 received --- 127.0.0.1 ping statistics ---
15:40:14 received 3 packets transmitted, 3 received, 0% packet loss, time 3999ms
15:40:14 received rtt min/avg/max/mdev = 0.035/0.038/0.041/0.005 ms