“UnicodeEncodeError: 'ascii' codec can't encode character” in Python3

Question

I'm fetching JSON with Requests from an API (using Python 3.5) and when I'm trying to print (or use) the JSON, either by response.text, json.loads(...) or response.json(), I get an UnicodeEncodeError.

print(response.text)
UnicodeEncodeError: 'ascii' codec can't encode character '\xc5' in position 676: ordinal not in range(128)

The JSON contains an array of dictionaries with country names and some of them contain special characters, eg: (just one dictionary in the binary array for example)

b'[{\n "name" : "\xc3\x85land Islands"\n}]

I have no idea why there is an encoding problem and also why "ascii" is used when Requests detects an UTF-8 encoding (and even by setting it manually to UTF-8 doesn't change anything).

Edit2: The problem was Microsoft Visual Studio Code 1.4. It wasn't able to print the characters.

Answer 1

If your code is running within VS, then it sounds that Python can't work out the encoding of the inbuilt console, so defaults to ASCII. If you try to print any non-ASCII then Python throws an error rather printing text that won't display.

You can force Python's encoding by using the PYTHONIOENCODING environment variable. Set it within the run configuration for the script.

Depending on Visual Studio's console, you may get away with:

PYTHONIOENCODING=utf-8

or you may have to use a typical 8bit charset like:

PYTHONIOENCODING=windows-1252