如何使用不同编码的 utf-8 和 Iso-8 在 Python 上读取多个文件

Question

I am still new to python and I am trying to read multiple files on a loop in python to calculate number of delimiters in each file however I have different encodings 'utf-8' and 'iso-' encoding I am not sure how to write my code with the condition if it's utf-8 then read it if not 'ISO-8' because I receive this error

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 1707: invalid continuation byte

And this is how my code looks like:

Thank you in advance!!

def number_delimiter_by_file(file):
    fin = open(file, encoding="utf8") 
    with fin as fp:
        lines = fp.readlines()
        i = 0
        a=[]
        for line in lines:
            i = i+1
            delimiter = line.count(";")
            a.append(delimiter)
        b = list(Counter(a).items())
    return(file, b)

#i apply the function on my list of files

for file in all_filenames:
    print(number_delimiter_by_file(file))

Answer 1

You have at least 2 ways to go past this error.

1. Ignore the actual encoding and only use ISO-8859-1 (aka Latin1). This encoding translates any byte on file into the Unicode character having that value. Because of that, it can read without error any file, but if the file uses a different encoding, some characters could be wrong. It should be enough to detect semicolons ( ; ) because they ASCII code (and unicode character code) is 0x22.

2. Use errors='ignore' when opening the file. In that mode, any offending character will be silently ignored. You will probably end with missing characters, but it should be enough to count semicolons characters.

Fortunately, all ISO-8859-x encodings are ASCII extension, so all ASCII characters have same encoding in any of them...