CSV 一列编码错误。 不能 pandas.read_csv

Question

I have a CSV file that contains several columns. One of those columns is corrupted, bad encoded. The column named title has characters of all kinds of languages: French, Italy, etc...

num | ratio |  title   | ...
 1     1.2     🥶2ï
 2     2.5     djije
 3     4.1     abc
...    ...     ...

When I try to read the file pandas.read_csv('myFile.csv') I receive the following error:

'utf-8' codec can't decode byte 0xcf in position 3: invalid continuation byte

How can I read the csv file with pandas and leaving the title column blank or giving it some default value if it can't be read?

Answer 1

If your file contains mixed encodings, you can read it into memory as binary, or as a hack, open it as Latin-1 and then decode the Title field individually on each line.

If the majority of the data is encoded as UTF-8, you can attempt to decode it with

title.encode('latin-1').decode('utf-8 )

but fall back and keep it in Latin-1, or replace it with some sort of error message, if decoding fails.

I'm not a Pandas person, but quick googling gets me something like

import pandas as pd

df = pd.read_csv('myFile.csv', encoding='latin-1')

def attempt_decode(x):
    try:
        return x.encode('latin-1').decode('utf-8')
    except UnicodeDecodeError:
        return 'Unable to decode: %s' % x)

df['Title'] = df['Title'].apply(attempt_decode)

Latin-1 has the unique property that every input byte corresponds exactly to the Unicode code point with the same value, so you never get a decoding error (but, obviously, mojibake if the correct encoding is something else, and you fail to correct it).

Answer 2

If you want to exclude the column title alone, read all the columns and drop the column title .

Eg.

df = pd.read_csv('filename')

df = data.drop('title', axis = 1)

To give the column a default value, use:

df['title'] = 0 #(the value you want to provide by default)

or use:

df['title'] = np.nan 

to fill the column with null values.

Hope that answers your question!