Why is the output of print in python2 and python3 different with the same string?

Question

In python2:

$ python2 -c 'print "\x08\x04\x87\x18"' | hexdump -C
00000000  08 04 87 18 0a                                    |.....|
00000005

In python3:

$ python3 -c 'print("\x08\x04\x87\x18")' | hexdump -C
00000000  08 04 c2 87 18 0a                                 |......|
00000006

Why does it have the byte "\\xc2" here?

Edit :

I think when the string have a non-ascii character, python3 will append the byte "\\xc2" to the string. (as @Ashraful Islam said)

So how can I avoid this in python3?

Answer 1

Consider the following snippet of code:

import sys
for i in range(128, 256):
    sys.stdout.write(chr(i))

Run this with Python 2 and look at the result with hexdump -C :

00000000  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|

Et cetera. No surprises; 128 bytes from 0x80 to 0xff .

Do the same with Python 3:

00000000  c2 80 c2 81 c2 82 c2 83  c2 84 c2 85 c2 86 c2 87  |................|
...
00000070  c2 b8 c2 b9 c2 ba c2 bb  c2 bc c2 bd c2 be c2 bf  |................|
00000080  c3 80 c3 81 c3 82 c3 83  c3 84 c3 85 c3 86 c3 87  |................|
...
000000f0  c3 b8 c3 b9 c3 ba c3 bb  c3 bc c3 bd c3 be c3 bf  |................|

To summarize:

• Everything from 0x80 to 0xbf has 0xc2 prepended.
• Everything from 0xc0 to 0xff has bit 6 set to zero and has 0xc3 prepended.

So, what's going on here?

In Python 2, strings are ASCII and no conversion is done. Tell it to write something outside the 0-127 ASCII range, it says “okey-doke!” and just writes those bytes. Simple.

In Python 3, strings are Unicode . When non-ASCII characters are written, they must be encoded in some way. The default encoding is UTF-8.

So, how are these values encoded in UTF-8?

Code points from 0x80 to 0x7ff are encoded as follows:

110vvvvv 10vvvvvv

Where the 11 v characters are the bits of the code point.

Thus:

0x80                 hex
1000 0000            8-bit binary
000 1000 0000        11-bit binary
00010 000000         divide into vvvvv vvvvvv
11000010 10000000    resulting UTF-8 octets in binary
0xc2 0x80            resulting UTF-8 octets in hex

0xc0                 hex
1100 0000            8-bit binary
000 1100 0000        11-bit binary
00011 000000         divide into vvvvv vvvvvv
11000011 10000000    resulting UTF-8 octets in binary
0xc3 0x80            resulting UTF-8 octets in hex

So that's why you're getting a c2 before 87 .

How to avoid all this in Python 3? Use the bytes type.

Answer 2

Python 2's default string type is byte strings. Byte strings are written "abc" while Unicode strings are written u"abc" .

Python 3's default string type is Unicode strings. Byte strings are written as b"abc" while Unicode strings are written "abc" ( u"abc" still works, too). since there are millions of Unicode characters, printing them as bytes requires an encoding ( UTF-8 in your case) which requires multiple bytes per code point.

First use a byte string in Python 3 to get the same Python 2 type. Then, because Python 3's print expects Unicode strings, use sys.stdout.buffer.write to write to the raw stdout interface, which expects byte strings.

python3 -c 'import sys; sys.stdout.buffer.write(b"\x08\x04\x87\x18")'

Note that if writing to a file, there are similar issues. For no encoding translation, open files in binary mode 'wb' and write byte strings.