IOError: [Errno 32] Broken pipe

Question

I usually observe this while running my automation scripts using paramiko ssh module in python. While executing some commands, it fails with the following error. Observed this error in tcl scripts as well, so realized it is not specific to a language

IOError: [Errno 32] Broken pipe

And we have observed it in while writing the steam of output to a file as well as below

file_handle.write(line_data)

We can handle the exception and add a retry block (Ref: IOError: [Errno 32] Broken pipe: Python )

But I am curios to know why is it happening at the first place so that I can take necessary precautions before running my job.

My findings resulted in "network drop" or "recipient system not responding". But I am not really convinced with those points. Please let me know the root cause

Answer 1

Broken pipe simply means that the receiving end of a pipe or a socket has already closed the connection. For example, consider this:

% python3 -c 'print("hello\n" * 2' | python3 -c 'import os; os.write(1, os.read(0, 6))'
hello

No error happens, because the pipe has a buffer that holds the excess data.

The buffer is nowadays 64Ki by default in Linux, so

% python3 -c 'print("h" * 65535)' | python3 -c 'import os; os.write(1, os.read(0, 5))'
hhhhh

but

% python3 -c 'print("h" * 65536)' | python3 -c 'import os; os.write(1, os.read(0, 5))'
hhhhhhException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

Here 65536 characters and the newline were being written; and the pipe error occurred only when the stream was being flushed at the end of the Python program.

---

To avoid the error, any script that reads the input must always consume all available input until end of file occurs, or the data-producing script must be careful enough to not produce more than expected.