Python struct.pack is not consistent

Question

I am using struct.pack to return me a bytes object. However, it is inconsistent.

print(struct.pack('BbbB', 0x1B, -2, 1, 0)) #returns b'\x1b\xfe\x01\x00' GOOD
print(struct.pack('BbbB', 0x0F, -2, 1, 0)) #returns b'\x0f\xfe\x01\x00' GOOD
print(struct.pack('BbbB', 0x0C, -2, 1, 0)) #returns b'\x0c\xfe\x01\x00' GOOD
print(struct.pack('BbbB', 0x35, -2, 1, 0)) #returns b'5\xfe\x01\x00'    ???
print(struct.pack('BbbB', 0x21, -2, 1, 0)) #returns b'!\xfe\x01\x00'    ???

If I look at ASCII character chart , I can see why it became "5" and "!" in the last two examples, but shouldnt it have returned me b' \\0x35 \\xfe\\x01\\x00' and b' \\0x21 \\xfe\\x01\\x00' instead of the above result? What am I missing here?

I am using Python 3.7.0.

Answer 1

Take a look at this:

>>> b'\x35\xfe\x01\x00'
b'5\xfe\x01\x00'
>>> b'\x35\xfe\x01\x00' == b'5\xfe\x01\x00'
True
>>> b'\x35'
b'5'
>>> b'\x35' == b'5'
True
>>> b'\x35'[0]
53
>>> b'5'[0]
53

Both b'\\x35' and b'5' have the same byte value 0x35. It is just that they are both two different representations of the same value.

It is very common that there are multiple ways to write a literal value for the same actual value. For example, the decimal value 53 can be represented with 53 , 0x35 , 0o65 , or 0b110101 . That are all number literals for the same int object.

And just like that b'\\x35 and b'5' are just two bytestring literals for the same bytes object.

When you print out objects, Python internally calls str() on the objects to make sure that the objects are printable. For bytes objects, this means that a bytes string literal is being returned. And since there are multiple ways to represent that object, Python has to decide on one representation.

For bytes, the rule is simply that whenever a byte can be represented as a printable ASCII character, that character is being used instead of an escape sequence. This generally allows you to read byte strings that represent ASCII data. For example b'foo' is a lot more readable than b'\\x66\\x6f\\x6f' (but means the same thing).

If you need to get a consistent string representation of a bytes object, what you can do is explicitly convert it into a hex string. For example by calling bytes.hex :

>>> b'5\xfe\x01\x00'.hex()
'35fe0100'

Of course, this is now no longer binary data, but a string representing binary data, encoded as a hex string. If you want to actually transmit the data, for example over the network or inside a file, you will want to keep it a bytes object and just use that.

Answer 2

It's not pack that is inconsistent. It's the print function that's confusing you.

Both '!' (ascii value 33, hexadecimal 0x21) and '5' (ascii value 53, hexadecimal 0x35) are printable characters, so they're just printed normally. The other bytes are not printable though, so python resorts to printing their hexadecimal representation so you can at least see something that makes sense.

The byte values are what you expect them to be, don't worry.