Utf-8 on windows python

Question

I have html file to read parse etc, it's encode on unicode (I saw it with the notepad) but when I tried

infile = open("path", "r") 
infile.read()

it fails and I had the famous error :

UnicodeEncodeError: 'charmap' codec can't encode characters in position xx: character maps to undefined

So for test I tried to copy paste the contain of the file in a new one and save it in utf-8 and then tried to open it with codecs like this :

inFile = codecs.open("path", "r", encoding="utf-8")
outputStream = inFile.read()

But I get this error message :

UnicodeEncodeError : 'charmap' codec can't encode character u'\' in position 0: charcater maps to undefined

I really don't understand because I was created this file in utf8.

Answer 1

UnicodeEncodeError suggests that the code fails while encoding Unicode text to bytes ie, your actual code tries to print to Windows console. See Python, Unicode, and the Windows console .

---

The link above fixes UnicodeEncodeError . The next issue is to find out what character encoding is used by the text in your "path" file. If notepad.exe shows the text correctly then it means that it is either encoded using locale.getprefferedencoding(False) (something like cp1252 on Windows) or the file has BOM .

If you are sure that the encoding is utf-8 then pass it to open() directly. Don't use codecs.open() :

with open('path', encoding='utf-8') as file:
    html = file.read()

Sometimes, the input may contain text encoded using multiple (inconsistent) encodings eg, smart quotes may be encoded using cp1252 while the rest of html is utf-8 -- you could fix it using bs4.UnicodeDammit . See also A good way to get the charset/encoding of an HTTP response in Python

Answer 2

In anticipation of the OP to update question to reflect the actual problem, the issue is caused by the encoding of the terminal not being defined.

The Windows console is notoriously poor when it comes to Unicode support. For ultimate support, see https://pypi.python.org/pypi/win_unicode_console . Essentially, install "win_unicode_console" ( pip install win_unicode_console ). Then at the top of your code:

import win_unicode_console
win_unicode_console.enable()

You may also need to use a suitable font - See https://stackoverflow.com/a/5750227/1554386

As you're using an input with a UTF-8 BOM, you should use the utf_8_sig codec so that the BOM is stripped before working with the contents.

As this is Python 3, you don't need to use the codecs module to set encoding when using open() .

Putting it together it would look like:

import win_unicode_console
win_unicode_console.enable()

infile = open("path", "r", encoding="utf_8_sig")

Answer 3

The original file probably uses utf-16 (Windows uses the term UNICODE for that encoding).

UTF-8 encoded files on Windows normally starts with a magic number b"\\xef\\xbb\\xbf" (the UTF-8 encoding of U+FEFF) so applications reading that file know it was saved as UTF-8 and not some ANSI code page. utf8-sig which will automatically discard that character.