在读取模式下读取二进制文件Python 3-在Windows上通过，在Linux上失败

Question

I am executing this piece of code against

Python on Windows

'3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)]'

and

Python on Linux

'3.6.6 (default, Mar 29 2019, 00:03:27) \\n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]'

The code writes some bytes into a file using wb mode and then reads it as r plain text. I understand that I should be reading as bytes ( rb ), but I am curious why does it break on Linux while passing on Windows?

import os
import tempfile
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, 'write_file')

expected_bytes = bytearray([123, 3, 255, 0, 100])
with open(temp_file, 'wb') as fh:
    fh.write(expected_bytes)

with open(temp_file, 'r', newline='') as fh:
    actual = fh.read()

Exception raised on Linux:

Traceback (most recent call last):
  File "<input>", line 11, in <module>
  File "/home/.../lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 2: invalid start byte

Getting default system encoding (with sys.getdefaultencoding() ) shows 'utf-8' on both machines.

Answer 1

When opening a file in text mode, so with 'rt' (where both 'r' and 't' are the default), everything you read from the file gets transparently decoded on the fly and returned as str objects, as explained in Text I/O .

You can force the encoding to use when opening the file, like:

f = open("myfile.txt", "r", encoding="utf-8")

As explained in the documenation for open :

The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any text encoding supported by Python can be used. See the codecs module for the list of supported encodings.

(Note that sys.getdefaultencoding() is something unrelated: it returns the name of the current default string encoding used by the Unicode implementation)

As you stated in the comments, on your system, locale.getpreferredencoding() gives 'cp1252' on Windows and 'UTF-8' on Linux.

CP-1252 is a single byte encoding in which each byte corresponds to a character. So, whatever file you read, the data it contains can be turned into a string.

UTF-8 , though, uses a variable width encoding in which not all sequences of bytes are valid and represent a character. That's why trying to read your file on your Linux system failed when some byte couldn't be decoded.

Answer 2

If you have written the file out as bytes, you should read it in as bytes.

f = open("myfile.txt", "rb")

If you read it in as text (using "r" or "rt" ) then an attempt will be made to decode it into Unicode. What encoding is used by default is platform-dependent. But you clearly don't want it decoded at all.