我不断收到 UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1: invalid start byte

Question

I'm trying to read monthly csv file but for some reason, I keep getting this error.

This is my code below.

df = pd.DataFrame()
 
for file in os.listdir("Performance_Data"):
    if file.endswith(".csv"):
        df = pd.concat([df , pd.read_csv(os.path.join("Performance_Data", file))], axis=0 )
        
df.head()

What do I do?

Answer 1

Pandas assumes by default that your file is encoded in UTF-8. Your file is encoded in Windows-1252. You can tell Pandas to use this encoding by

pd.read_csv(os.path.join("Performance_Data", file), encoding='cp1252')

Detecting the encoding of a file automatically is a bit tricky, but you can use a package called "chardet". For your code, it could look like this:

import os

import chardet
import pandas as pd

df = pd.DataFrame()

for file in os.listdir("Performance_Data"):
    if file.endswith(".csv"):
        with open(file, "rb") as fp:
            encoding = chardet.detect(fp.read())["encoding"]
        df = pd.concat(
            [
                df,
                pd.read_csv(os.path.join("Performance_Data", file), encoding=encoding),
            ],
            axis=0,
        )

df.head()

References

• Pandas read_csv documentation.
• UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 377826: invalid start byte , a relevant earlier question on Stack Overflow.