read on a pipe blocks until program running at end of pipe terminates - Windows

Question

I have a sample program that outputs a line of text every second. In the test program below, this program writes some text to stdout then waits 1 second and repeats 20 times.

I have another program which uses popen ( _popen on Windows) to open a pipe for reading from the program. I then use fgets to read data. The problem I have is that the fgets blocks until the program terminates. Then I get all the output, all 20 lines, in one go. I want to get the output a line at a time, then ok for fgets to block until next line ready. The reason is I plan to use this on a program that will be constantly running, outputting text, eg like the use of tail .

If I run this code example on a program that outputs some text all in one go and exits then it works fine.

Why does fgets block? The test program does print some text immediately, so why doesn't fgets read this first line of text immediately?

Here is the code:

#include <stdio.h>
#include <windows.h>

void execute(const char* cmd) {
    char buffer[128] = { 0 };
    FILE* pipe = _popen(cmd, "r");

    if (!pipe) {
        printf("popen() failed!\n");
        return;
    }

    while (!feof(pipe)) {
        if (fgets(buffer, 128, pipe) != nullptr)
            printf("%s", buffer);
    }

    int rc = _pclose(pipe);

    if (rc != EXIT_SUCCESS) { // return code not 0
        printf("pclose exit failure: %d\n", rc);
    }
}


int main(int argc, char* argv[]) {
    if (argc != 2) {
        printf("Usage: pipe_test.exe <program>\n");
        exit(1);
    }

    execute(argv[1]);
}

The program run, helloworld.exe :

#include <stdio.h>
#include <windows.h>

int main() {

    for (int i = 0; i < 20; i++) {
        printf("Hello World %d\n", i);
        Sleep(1000);
    }
}

Answer 1

Why does fgets block?

Because it's waiting for the children to output something.

The test program does print some text immediately, so why doesn't fgets read this first line of text immediately?

It actually does not print text immediately. The problem here, as @Barmar notices, is that writing to a pipe is buffered (and not line buffered) by the C standard library implementation. This buffering happens in your child program ( helloworld ), not in your parent program ( pipe_test ).

From your parent program, you have no control over what the children spawned through popen() will do, therefore if the child output is buffered like in this case, the only thing you can do (without modifying the child's code) is to wait until the buffer is flushed to the pipe.

In order to get the output sooner, you would have to modify the children's code to manually call fflush() or use setvbuf() to disable buffering:

int main() {
    setvbuf(stdout, NULL, _IONBF, 0); // Disable buffering on stdout.

    for (int i = 0; i < 20; i++) {
        printf("Hello World %d\n", i);
        Sleep(1000);
    }
}

There's really not much else you can do.