Interprocess communication, reading from multiple children stdout

Question

I'm trying to write a custom shell-like program, where multiple commands can be executed concurrently. For a single command this is not much complicated. However, when I try to concurrently execute multiple commands (each one in a separate child) and capture their stdout I'm having a problem.

What I tried so far is this under my shell application I have two functions to run the commands concurrently, execute() takes multiple commands and for each of the commands it fork() a child process to execute the command, subprocess() takes 1 cmd and executes it.

void execute(std::vector<std::string> cmds) {
    int fds[2];
    pipe(fds);
    std::pair<pid_t, int> sp;

    for (int i = 0; i < cmds.size(); i++) {
        std::pair<pid_t, int> sp = this->subprocess(cmds[i], fds);
    }

    
    // wait for all children
    while (wait(NULL) > 0);
    close(sp.second);
}

std::pair<pid_t, int> subprocess(std::string &cmd, int *fds) {
    std::pair<pid_t, int> process = std::make_pair(fork(), fds[0]);
    if (process.first == 0) {
        close(fds[0]); // no reading
        dup2(fds[1], STDIN_FILENO);
        close(fds[1]);
        char *argv[] = {"/bin/sh", "-c", cmd.data(), NULL};
        execvp(argv[0], argv);
        exit(0);
    }
    close(fds[1]); // only reading
    return process;
} 

The problem here is, when I execute multiple commands on my custom shell (not diving into spesifics here, but it will call execute() at some point.) if I use STDIN_FILENO as above to capture child process stdout, it keeps writing to shell's stdin forever what the captured output is, for example

if the input commands are

echo im done, yet?
echo nope
echo maybe

then, in writing to STDIN_FILENO case, the output is like (where >>> ) is my marker for user input.

im done, yet?
nope
maybe
>>> nope
maybe
im done, yet?
>>> im done, yet?
nope
maybe

in writing to STDOUT_FILENO case, it seems it's ignoring one of the commands (probably the first child), I'm not sure why?

maybe
nope
>>> maybe
nope
>>> nope
maybe
>>> maybe
nope
>>> nope

So, potential things I thought are in my shell I'm using std::cin >>... for user input in a while loop ofc, this may somehow conflict with stdin case. On the other hand, in the main process (parent) I'm waiting for all children to exit, so children somehow is not exiting, but child should die off after execvp, right? Moreover, I close the reading end in the main process close(sp.second) . At this point, I'm not sure why this case happens?

Should I not use pipe() for a process like this? If I use a temp file to redirect stdout of child process, would everything be fine? and if so, can you please explain why?

Answer 1

There are multiple, fundamental, conceptual problems in the shown code.

std::pair<pid_t, int> sp;

This declares a new std::pair object. So far so good.

std::pair<pid_t, int> sp = this->subprocess(cmds[i], fds);

This declares a new std::pair object inside the for loop. It just happens to have the same name as the sp object at the function scope. But it's a different object that has nothing to do, whatsoever, with it. That's how C++ works: when you declare an object inside an inner scope, inside an if statement, a for loop, or anything that's stuffed inside another pair of { ... } you end up declaring a new object. Whether its name happens to be the same as another name that's been declared in a larger scope, it's immaterial. It's a new object.

    // wait for all children
    while (wait(NULL) > 0);
    close(sp.second);

There are two separate problems here.

1. For starters, if we've been paying attention: this sp object has not been initialized to anything.

2. If the goal here is to read from the children, that part is completely missing, and that should be done before waiting for the child processes to exit. If, as the described goal is here, the child processes are going to be writing to this pipe the pipe should be read from. Otherwise if nothing is being read from the pipe: the pipe's internal buffer is limited, and if the child processes fill up the pipe they'll be blocked, waiting for the pipe to be read from. But the parent process is waiting for the child processes to exist, so everything will hang.

Finally, it is also unclear why the pipe's file descriptor is getting passed to the same function, only to return a std::pair with the same file descriptor. The std::pair serves no useful purpose in the shown code, so it's likely that there's also more code that's not shown here, where this is put to use.

At least all of the above problems must be fixed in order for the shown code to work correctly. If there's other code that's not shown, it may or may not have additional issues, as well.