UnicodeEncodeError: 'ascii' codec can't encode characters ordinal not in range(128)

Question

I can't read the word Curaçao from a text file. What am I doing wrong?

I have written a text file that contains the word "Curaçao". The encoding on the editor (vim) is latin1.

This python program reads the file:

import sys

with open ('foo.txt', 'r', encoding='latin1') as f:
    print('f:', f.encoding)
    print('stdout:', sys.stdout.encoding)
    for i in f:
        print(i)

And when I run it I get this...

sundev19:/home/jgalloway12/code/wdPhone $ python3 CountryFix.py
f: latin1
stdout: 646
Traceback (most recent call last):
  File "CountryFix.py", line 11, in <module>
    print(i)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe7' in position 4: ordinal not in range(128)

Here is the file's contents in binary.

0000000: 4375 7261 e761 6f0a                      Cura.ao.

EDIT: The "real" problem I am trying to solve here is reading an Excel 2010 exported CSV which contains country names.

Fixed the file to be encoded in Latin1. Program now prints locale.

Answer 1

The problem here isn't the file, but the output stream.

For whatever reason, python has detected your stdout encoding as US-ASCII when you really want something more (utf-8, latin1, etc.).

Your options are:

Trick it into believing a different encoding (on linux you can do this with LANG=en_US.UTF-8 , however I assume you're on windows and I don't recall how to trick python on windows in this way :)).

Write your response to a file:

with open('output.txt', 'w', encoding='latin1') as f:
    ...

Or write to the stdout bytestream:

import sys
sys.stdout.buffer.write(i.encode('latin1'))

Answer 2

Since you are printing the lines and python print function doesn't use of the encoding of open() function it tries to encode your string with it's default encoding which is ASCII. So you need to define a costume encoding for your unicode when you want to print it.

You can use str.encode() method with a proper encocding for print.