在python3中正确使用unicode字符-强制utf-8编码

Question

I'm going crazy here. The internet and this SO question tell me that in python 3.x, the default encoding is UTF-8. In addition to that, my system's default encoding is UTF-8. In addition to that, I have # -*- coding: utf-8 -*- at the top of my python 3.5 file.

Still, python is using ascii:

# -*- coding: utf-8 -*-
mystring = "Ⓐ"
print(mystring)

Greets me with:

SyntaxError: 'ascii' codec can't decode byte 0xe2 in position 7: ordinal not in range(128)

I've also tried this: print(mystring.encode("utf-8")) and .decode("utf-8") - Same thing.

What am I missing here? How do I force python to stop using ascii encoding?

---

Edit: I know that it seems weird to complain about position 7 with a one character string, but this is my actual MCVE and the exact output I'm getting. The above is using python shell, the below is in a script. Both use python 3.5.2 .

---

Edit: Since I figured it might be relevant: The string I'm getting comes from an external application and is not hardcoded, so I need a way to get that utf-8 string and save it into a file. The above is just a minimalized and generalized example. Here is my real-life code:

# the variables being a string that might contain unicode characters
mystring = "username: " + fromuser + " | printname: " + fromname
with open("myfile.txt", "a") as myfile:
  myfile.write(mystring + "\n")

Answer 1

In Python3 all strings are unicode, so the problem you're having is likely due to your locale settings not being correct. The Python3 interpreter looks to use the locale environment variables and if it cannot find them it emulates basic ASCII

From locale.py:

except ImportError:

    # Locale emulation

    CHAR_MAX = 127
    LC_ALL = 6
    LC_COLLATE = 3
    LC_CTYPE = 0
    LC_MESSAGES = 5
    LC_MONETARY = 4
    LC_NUMERIC = 1
    LC_TIME = 2
    Error = ValueError

Double check the locale on your shell from which you are executing . Here are a few work arounds you can try to see if they get you working before you go through the task of getting your env setup correctly.

1) Validate UTF-8 locale or language files are installed (see link above)

2) Try adding this to the top of your script

#!/usr/bin/env LC_ALL=en_US.UTF-8 /usr/local/bin/python3
print('カタカナ')

or

#!/usr/bin/env LANG=en_US.UTF-8 /usr/local/bin/python3
print('カタカナ')

Or export shell variables before executing the Python interpreter

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
python3
>>> print('カタカナ')

Sorry I cannot be more specific, as these settings are platform and OS specific. You can forcefully attempt to set the locale in Python directly using the locale module , but I don't recommend that, and it won't help if they are not installed.

Hope that helps.

Answer 2

What's new in Python 3.0 says:

All text is Unicode; however encoded Unicode is represented as binary data

If you want to try outputting utf-8, here's an example:

b'\x41'.decode("utf-8", "strict")

If you'd like to use unicode in a string literal, use the unicode escape and its coded representation. For your example:

print("\u24B6")