debugging stalled write() to std::cout from libc.so.6

Question

I have just observed an extremely rare occurrence of stall in a binary.

Attaching gdb and switching to the relevant thread gives

(gdb) bt
#0  0x000000330a4db79d in write () from /lib64/libc.so.6
#1  0x000000330a471dd3 in _IO_new_file_write () from /lib64/libc.so.6
#2  0x000000330a473385 in _IO_new_do_write () from /lib64/libc.so.6
#3  0x000000330a4726df in _IO_new_file_overflow () from /lib64/libc.so.6
#4  0x000000330a46f437 in putc () from /lib64/libc.so.6
#5  0x00007f92d94864ea in sputc (__c=10 '\n', this=<optimized out>) at [..omitted..]/gcc-4.9.0-objdir/x86_64-linux-gnu/libstdc++-v3/include/streambuf:434
#6  std::ostream::put (this=0x3171f40 <std::cout>, __c=<optimized out>) at [..omitted..]/gcc-4.9.0-objdir/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:163

and strace gives

Process 14702 attached - interrupt to quit
write(1, "[...some string...]", 31

It has been there forever, writing to std::cout. What other information should I gather? How should one get to the bottom of this?

EDIT: The binary is invoked by a python system, very deep inside, it's doing

pipe = subprocess.Popen(
  command,
  shell=True,
  stdout=subprocess.PIPE,
  stderr=subprocess.STDOUT
)
output = pipe.stdout.read()
pipe.wait()

Answer 1

If cout is redirected to a pipe, then output will block if the process on the other end of the pipe doesn't read fast enough. You need to decide whether the C++ process should be constructed to continue to make forward progress even if writes to cout block or whether the other end of the pipe must read from the pipe. Figure out what's broken and fix it.