Python 3字节在CGI脚本中使用非ASCII字符进行解码

Question

I have a very short sample code:

print("Content-Type: text/plain; charset=utf-8")
print("Access-Control-Allow-Origin: *")
print()

x = 'Chloë'.encode()
print(x)
print(x.decode())

Notice non Ascii ë , which is the source of all problems.

Calling the script in bash using python3 ./test.py produces following (correct) input:

Content-Type: text/plain; charset=utf-8
Access-Control-Allow-Origin: *

b'Chlo\xc3\xab'
Chloë

However calling it from the browser, the last line is not present (headers of course aren't visible, but they are present). So the only visible part is:

b'Chlo\xc3\xab'

Do you know, where could be a problem?

Answer 1

You are printing Unicode to the sys.stdout handle (which is the default file object print() writes to). That object then has to encode your data again, but it has to do so based on the environment that it is connected to.

When you run python3 ./test.py then you connected to your terminal or console, and it is usually configured to tell scripts what codec is appropriate. On POSIX systems (Linux, Mac) you can run the locale command to see what that configuration is. In your console locale there is no problem displaying a non-ASCII codepoint like ë .

But when running as a CGI script connected to a webserver, there is no such language configuration present, and Python almost certainly has fallen back to the lowest common denominator instead: ASCII. And when this is the case, trying to print non-Unicode text will result in an exception:

$ LC_ALL="en_US.UTF-8" python3 -c "print(b'Chlo\xc3\xab'.decode())"
Chloë
$ LC_ALL="C" python3 -c "print(b'Chlo\xc3\xab'.decode())"  # C => "no locale set"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xeb' in position 4: ordinal not in range(128)

Because the exception takes place only after producing headers and all the other output, you don't see an HTTP error code. The exception should have been logged in your server error logs, however.

If your script is to output UTF-8 to the browser, as configured in the Content-Type header you emit, replace sys.stdout to force that codec:

import sys
from io import TextIOWrapper

sys.stdout = TextIOWrapper(sys.stdout.buffer.detach(), encoding='utf8')

In Python 3, text files like those used for the sys.stdout stream, contain a buffer object, which in turn contains a binary file object that takes care of the actual binary data writing. The outer text file object is only responsible for encoding on write, really. The above replaces that outer object with a different one that always encodes to UTF-8.