为什么在不同的操作系统和版本中打印这些值会给出不同的值？

Question

Why does printing these \\x values give different values in different OS and versions? Example:

print("A"*20+"\xef\xbe\xad\xde")

This gives different output in Python3 and 2 and in different platforms

In Microsoft's Windows:

Python2: AAAAAAAAAAAAAAAAAAAAï¾Þ

Python3: AAAAAAAAAAAAAAAAAAAAï¾Þ

In Kali:

Python2: AAAAAAAAAAAAAAAAAAAAﾭ

Python3: AAAAAAAAAAAAAAAAAAAAï¾Þ

UPDATE: What I want is the exact Python2 output but with Python3? I tried many things(encoding, decoding, byte conversion) but realised \\xde can't be decoded. Any other way to achieve what I want?

Answer 1

It is a question of encoding.

In Latin1 or Windows 1252 encoding, you have:

0xef -> ï (LATIN SMALL LETTER I WITH DIAERESIS)
0xbe -> ¾ (VULGAR FRACTION THREE QUARTERS)
0xad -> undefined and non printed in your examples
0xde -> Þ (LATIN CAPITAL LETTER THORN)

In utf-8 encoding, you have:

'\\xef\\xbe\\xad' -> u'\ﾭ' or 'ﾭ' (HALFWIDTH HANGUL LETTER RIEUL-SIOS) '\\xde' -> should raise an UnicodeDecodeError...

In Windows, Python2 or Python3 both use Windows 1252 code page (in your example). On Kali, Python2 sees the string as byte string and the terminal displays it in utf8, while Python3 assumes it already contains unicode character values and displays them directly.

As in Latin1 (and in Windows 1252 for all characters outside 0x80-0x9f) the byte code is the unicode value, that is enough to explain your outputs.

What to learn: be explicit whether strings contains unicode or bytes and beware of encodings!

Answer 2

To get consistent behavior on both Python 2 and Python 3, you'll need to be explicit about your intended output. If you want, AAAAAAAAAAAAAAAAAAAAﾭ , then the \\xde is garbage; if you want AAAAAAAAAAAAAAAAAAAAï¾Þ , the \\xad is garbage. Either way, the "solution" to printing what you've got is to explicitly use bytes literals and decode them with the desired encoding, ignoring errors. So to get AAAAAAAAAAAAAAAAAAAAﾭ (interpreting as UTF-8), you'd do:

print((b"A"*20+b"\xef\xbe\xad\xde").decode('utf-8', errors='ignore'))

while to get AAAAAAAAAAAAAAAAAAAAï¾Þ you'd do:

# cp1252 can be used instead of latin-1, depending on intent; they overlap in this case
print((b"A"*20+b"\xef\xbe\xad\xde").decode('latin-1', errors='ignore'))

Importantly, note the leading b on the literals; they're recognized and ignored on Python 2.7 (unless from __future__ unicode_literals is in effect, in which case they're needed just like in Python 3) and on Python 3, it makes the literals bytes literals (no special encoding assumed), rather than str literals, so you can decode in your desired encoding. Either way, you end up with raw bytes, which can then be decoded in the preferred encoding, with errors ignored.

Note that ignoring errors is usually going to be wrong; you're dropping data on the floor. 0xDEADBEEF isn't guaranteed to produce a useful byte string in any given encoding, and if that's not your real data, you're probably still risking errors by wanting to silently ignore undecodeable data.

If you want to write the raw bytes and let whatever is consuming stdout interpret them however it wants, you need to drop below the print level, since print on Python 3 is purely str based. To write the raw bytes on Python 3, you'd use sys.stdout.buffer ( sys.stdout is text based, sys.stdout.buffer is the underlying buffered byte-oriented stream it wraps); you'd also need to manually add the newline (if desired):

sys.stdout.buffer.write(b"A"*20+b"\xef\xbe\xad\xde\n")

vs. on Python 2 where stdout isn't an encoding wrapper:

sys.stdout.write(b"A"*20+b"\xef\xbe\xad\xde\n")

For portable code, you can get a "raw stdout" ahead of time and use that:

# Put this at the top of your file so you don't have to constantly recheck/reacquire
# Gets sys.stdout.buffer if it exists, sys.stdout otherwise
bstdout = getattr(sys.stdout, 'buffer', sys.stdout)

# Works on both Py2 and Py3
bstdout.write(b"A"*20+b"\xef\xbe\xad\xde\n")