python subprocess output issue with PIPE, when trying to capture free running outputs

Question

Input C Program test.c

#include <stdio.h>

int main(void)
{
    while(1)
    {
        printf("Hello World!\n");
        sleep(1);
    }
    return 0;
}

Compile with gcc gcc -g -o test test.c

Run the program to see its giving the output

./test 
Hello World!
Hello World!
Hello World!
^C

Run the program using subprocess to capture output

import subprocess
import shlex

cmd = "./test"
def run_command(command):
    cmds = shlex.split(cmd)
    print(cmds)
    proc = subprocess.Popen(cmds, stdout=subprocess.PIPE)
    try:
        outs, errs = proc.communicate(timeout=15)
    except subprocess.TimeoutExpired:
        proc.kill()
        outs, errs = proc.communicate()
    print(outs)
    
run_command(cmd)

Output is empty

python3 test.py 
['./test']
b''

Replace cmd = "./test" with cmd = "gdb -x load./test" where load is a file with run command

Output is just the gdb info, but no application output

python3 test.py 
['gdb', '-x', 'load', './test']
b'GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1\nCopyright (C) 2022 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\nType "show copying" and "show warranty" for details.\nThis GDB was configured as "x86_64-linux-gnu".\nType "show configuration" for configuration details.\nFor bug reporting instructions, please see:\n<https://www.gnu.org/software/gdb/bugs/>.\nFind the GDB manual and other documentation resources online at:\n    <http://www.gnu.org/software/gdb/documentation/>.\n\nFor help, type "help".\nType "apropos word" to search for commands related to "word"...\nReading symbols from ./test...\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n'

Answer 1

You are looking at a buffering issue. When a C program's standard output is an interactive device, output to it is line-buffered by default. Otherwise, including when the standard output is a pipe, the output to it is block-buffered.

With line-buffered output, each "Hello World.\n" is dispatched to the stream right away because of the trailing newline, With block-buffered output, it's all saved in a buffer inside the program until that fills. then that is sent all at once.

When run via your Python script, the C program writes only about 15 short lines (195 bytes) to its block-buffered standard output, not enough to fill its buffer. When the process is then killed via a signal, it terminates immediately without flushing the buffer, so the Python script never sees any of the output.

You can fix that by adding a fflush(stdout);immediately after the printf() . And that's probably analogous to what gdb is doing.