Structs unpack and pack changes data

Question

I am currently trying to pack some data in a script "p1.py", and then unpack it in another script "p2.py" via a pipe. The pipe communication works perfectly, and I'm able to send data between the two scripts. However, upon unpacking the data gets changed. In this case, I do this in script "p2.py" in order to pack the data and send it over to the "p1.py" script:

COMMAND_STRUCT = struct.Struct(">fB3s")

packed_data = struct.pack(">fB3s",29.1,1,b'3s')
print(packed_data)
#lateral_airspeed_input, drop_package_commanded_byte, _ = COMMAND_STRUCT.unpack(packed_data)
#print(lateral_airspeed_input)

As we can see, I print "packed_data" which is packed to be a float, a byte and then 3 bytes. Now if I simply print the packed data to my terminal, I get this output:

b'A\xe8\xcc\xcd\x013s\x00'

Which is simply the packed form of the 29.1,1 and 3 bytes. If I uncomment the last two lines, lateral_airspeed_input will be 29.1 -indicating that the unpacking and packing works, as I can pack the data and unpack and get the same data after unpacking.

Now, here's where it get interesting. When I bring in the other script, I do the same thing as seen here:

        cmd = pilot.stdout.read(COMMAND_STRUCT.size)
        #print(cmd)
        if len(cmd) != COMMAND_STRUCT.size:
            result = CRASHED  # The pilot process must have exited
            break
        lateral_airspeed_input, drop_package_commanded_byte, _ = COMMAND_STRUCT.unpack(cmd)
        print(lateral_airspeed_input)
        lateral_airspeed = max(-30.0, min(30.0, lateral_airspeed_input))
        drop_package_commanded = bool(drop_package_commanded_byte)

Here, the packed data is read onto the variable cmd, and then cmd is unpacked to the lateral_airspeed_input, drop_package_commanded_byte and the _ variables. Ideally, if we print lateral_airspeed_input we should be getting 29.1. However, I get 7.713289749049545e+20 upon printing it. However, if I just print cmd straight away, I get this:

b"b'A\xe8\" b'xcc\xcd\' b'x013s\x0' b"0'\r\n"

which looks similar to the packed data from before - when I was just using one py script. For some reason, the packed data gets all these extra, b" and 0\r\n things on it. I think this is messing up the communication and I was wondering if there's a way to avoid this. I'd rather avoid this than stripping the codes off the lines.

Thanks for the help. I am using subprocess pipes to manage the communication between the two scripts, and simply running the subprocess through my windows cmd.

Answer 1

I tried to put together a sample based on what you provided.

This is script p1.py:

import struct

COMMAND_STRUCT = struct.Struct(">fB3s")

packed_data = struct.pack(">fB3s",29.1,1,b'3s')
print(packed_data)

This is script p2.py:

import struct
import sys

COMMAND_STRUCT = struct.Struct(">fB3s")
cmd = sys.stdin.read(COMMAND_STRUCT.size)
#print(cmd)
if len(cmd) != COMMAND_STRUCT.size:
    sys.exit(1)
lateral_airspeed_input, drop_package_commanded_byte, _ = COMMAND_STRUCT.unpack(cmd)
print(lateral_airspeed_input)
lateral_airspeed = max(-30.0, min(30.0, lateral_airspeed_input))
drop_package_commanded = bool(drop_package_commanded_byte)

If I run it like this, I believe I get the expected result:

%> python2 p1.py | python2 p2.py

29.1000003815

That seems like the desired result. Can you elaborate on what's different in your scenario vs. this small example?

UPDATE

Following on your previous comment, I made a new script p3.py that looks like this:

import subprocess
import struct
import sys

COMMAND_STRUCT = struct.Struct(">fB3s")
pilot = subprocess.Popen(['python2','./p1.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd = pilot.stdout.read(COMMAND_STRUCT.size)
#cmd = sys.stdin.read(COMMAND_STRUCT.size)
#print(cmd)
if len(cmd) != COMMAND_STRUCT.size:
    sys.exit(1)
lateral_airspeed_input, drop_package_commanded_byte, _ = COMMAND_STRUCT.unpack(cmd)
print(lateral_airspeed_input)
lateral_airspeed = max(-30.0, min(30.0, lateral_airspeed_input))
drop_package_commanded = bool(drop_package_commanded_byte)

When I run it, I do still seem to get the desired result:

%> python2 p3.py

29.1000003815

So I must still be missing something here I think.

Answer 2

As already noted in the comments, your sender script is printing the representation of the string. Ie instead of the byte sequence 41 e8 cc cd... ( A.. ) you get 62 21 41... ( b"A\... ).

Also, print and stdout are by default working in text mode. Ie you will get in trouble with raw binary data. The documentation says that you can use the underlying buffer of stdio to send binary data:

import sys
sys.stdout.buffer.write(packed_data)
sys.stdout.flush()

You must use flush to actually send the data. Otherwise it would only be written when a newline character ( 0x0d ) is encountered.

Also, if you do this, make sure that the stdout is not actually printed to screen. Otherwise your terminal will be messed up rather quickly.

Vice versa, you should use binary reading in the receiver script: cmd = sys.stdin.buffer.read() .