在python 3.6中将字节转换为字符串

Question

I am trying to read and process a file. This wroks perfectly fine in Python2.7 but I can't get it working in Python 3. In Python 2.7 it works without providing any encoding whereas in Python 3 I have tried all combinations with and without encoding.

After deep diving, I found that the way content returned by read is different in both the versions.

Code in Python 2.7 that works:

>>> f = open('resource.cgn', 'r')
>>> content = f.read()
>>> type(content)
<type 'str'>
>>> content[0:20]
'\x04#lwq \x7f`g \xa0\x03\xa3,ess to'
>>> content[0]
'\x04'

However in Python 3:

>>> f = open('resource.cgn','r')
>>> content = f.read()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec cant decode byte 0xa0 in position 10: ordinal not in range(128)
>>> f = open('resource.cgn','rb')
>>> content = f.read()
>>> type(content)                   
<class 'bytes'>
>>> content[0:20]
b'\x04#lwq \x7f`g \xa0\x03\xa3,ess to'
>>> content[0]
4
>>> content.decode('utf8')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 10: 
invalid start byte

I would like to get the same output as in Python 2.7. The content should be of type string and content[0] should be str '\\x04' and not int 4

Any pointers on how can I get this? I have tried encodings without any success.

Answer 1

3.X's str is now 2.X's unicode by default and file objects opened in text mode in 3.X attempt to decode and encode when your files are read from or written to, respectively. str of 2.X is now bytes in 3.X. There's really very minor differences between 3.X bytes and 2.X's str both essentially hold 8-bit text.

Here's a simple trick to convert b'\\x04#lwq \\x7f`g \\xa0\\x03\\xa3,ess to' to str in 3.X:

>>> content = ''.join(chr(x) for x in b'\x04#lwq \x7f`g \xa0\x03\xa3,ess to')
>>> content
'\x04#lwq \x7f`g \xa0\x03£,ess to'
>>> content[0]
'\x04

Decoding the bytes string fails because you have invalid UTF-8 character bytes, same for ASCII.

However, it's wise to mention that bytes is meant to process binary data and str is for Unicode strings only in 3.X. It's recommended then to use bytes instead of str for binary strings in 3.X:

>>> content = b'\x04#lwq \x7f`g \xa0\x03\xa3,ess to'
>>> hex(content[0])
'0x4'