熊猫read_excel奇怪的错误：UnicodeDecodeError：'ascii'编解码器无法解码字节0xe2

Question

When I read an excel file in IPython (or Jupyter to be exact), the dataframe seems to be read ok, but I can't display it or work on textual columns on it (for instance, to merge with another excel when the key is the textual field), because I get a "

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2...

error

The strange part is that if I do something like:

for i in df['Textual Col Name']:
    print i

it prints all the values ok.

I've tried the different solutions offered to other similar questions here but nothing worked. I don't think there's a good answer on SO for when the df is read from an excel file.

Would love to get your help how to work out this problem and if possible also an explanation why all said and done I can still print the individual items just fine.

Thanks in advance!

Answer 1

You need to specify the encoding of the file, without having the file is is impossible to know how it is encoded but you can try a few and see which works if you don't know. encoding=tuf-8 , encoding=latin-1 or encoding=cp1252 in pd.read_excel .

Answer 2

Try using the method of bisection to isolate the problematic row:

import numpy as np
import pandas as pd

# substitute your df here
df = pd.DataFrame({'textcol':np.random.randint(10, size=[1000])})

def isokay(df):
    try:
        print(df)
    except UnicodeDecodeError:
        return False
    return True

i = 0
chunksize = len(df) // 2
while True:
    if isokay(df.iloc[i:i+chunksize]):
        i += chunksize
        if i > len(df):
            print('No error found')
            break
    else:
        if chunksize <= 1:
            # Problem occurs at row i
            print('Problem occurs on row {}'.format(i))
            print(df.iloc[i])
            break
        else:
            chunksize /= 2

The line print(df.iloc[i]) may result in an error. If so, you could consult the excel file to find out what data is contained on row i .