Python, Windows and a Unicode file

Question

I'm trying to execute a simple utility I wrote for Linux, which I thought it would be executed without problem on Windows. Wrong.

The script parses a simple file using the "re" module for regex. The problem is that the expression fails every time because Windows doesn't treat well the text file, which is UTF-8, because it contains things like áéíóú or ñ (it's in Spanish).

I've found a lot of stuff about printing text in Unicode format, but have found nothing about reading an Unicode line from a text file or using regex with Unicode on Windows. Thought you might shed some light on the subject.

Answer 1

open() uses locale.getpreferredencoding(False) encoding to decode a file. It is likely to be utf-8 on POSIX systems and it is something else on Windows eg, cp1252 .

If you know the text in the file is stored using utf-8 character encoding then pass it explicitly:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import io
import re

with io.open("filename.txt", encoding='utf-8') as file:
    for line in file:
        if re.search(u"(?u)unicode\s+pattern…", line):
            # found..