为什么在写入python子进程stdin管道时丢失数据？

Question

My python code looks like this:

def test():
    pipe = sp.Popen( ["test.sh"], stdin=sp.PIPE)
    data = "".join([chr((s)%17) for s in range(0,33)])
    os.write(pipe.stdin.fileno(), data)
    pipe.stdin.write("endoffile")

if __name__ == "__main__":
    test()

It calls the following simple bash shell script which simply writes stdin to a file (script is called test.sh)

#!/bin/bash
VALUE=$(cat)

echo "$VALUE" >> /tmp/test.txt

When I run the python code I expect test.txt to contain the values 0x01..0x10 two times, and after that the string "endoffile"

However here's a hexdump of the file:

0000000: 0102 0304 0506 0708 090a 0b0c 0d0e 0f10  ................
0000010: 0102 0304 0506 0708 090a 0b0c 0d0e 0f65  ...............e
0000020: 6e64 6f66 6669 6c65 0a                   ndoffile.

It appears that a byte is missing (0x10).

What am I missing here?

--- Update

Changing the test() function to:

def test():
    pipe = sp.Popen( ["test.sh"], stdin=sp.PIPE)
    data = "".join([chr((s)%16+1) for s in range(0,32)])
    os.write(pipe.stdin.fileno(), data)
    pipe.stdin.write("endoffile")

Seems to solve that. It seems to be related to having chr(0) sent to pipe.

Answer 1

range() is right side exclusive.

range(0, 33) is [0, ..., 32] , probably because this way you can range(0, len(sequence)) without off-by-one errors.

Since 32 % 17 == 15 == 0x0f , the byte '\\x10' you are expecting was never part of the list in the first place.

Edit 1: Also missing from the output are the zero characters '\\x00' . If you use VALUE=$(cat) the output of cat is subject to processing by the shell.

SingleUnix/POSIX seems to be silent on the matter. It is however clear, that you cannot have '\\0' as part of a shell variable's value (or name for that matter) since the Unix environment requires both to be C-style zero terminated strings . I actually would have expected the value of VALUE to be an empty string.

Edit 2 After some digging, I can say that at least the ash implementation ignores '\\0' processing backtick-supplied input. Input is read until EOF and null characters are explicitly skipped.

bash does the same and even has an explicit (even if commented out) warning associated with the event.