在熊猫数据框中查找和替换坏字符

Question

I'm getting stuck trying to get rid of bad characters in a pandas dataframe. This is an automated script that processes incoming data that needs to be saved in cp1252, and I want to be able to handle any problem characters on the fly by parsing the error. I don't care what they are replaced with. I've tried a million variations on this and can't get anywhere (this is python 3 pandas 25)

while True:
    try:
        print('saving')
        data.to_csv('total.csv', index=False, quoting=csv.QUOTE_ALL, encoding='cp1252')
        break
    except UnicodeEncodeError as e:
        print(e)
        badchar = re.search(r"character (.+?) in", str(e)).group(1)
        print('Found bad character, removing. . . ')
        uchar = u"{}".format(badchar)
        print(uchar)
        data = data.replace(uchar.encode('utf-8'), '')

Returns:

saving
'charmap' codec can't encode character '\u2264' in position 399: character maps to <undefined>
Found bad character, removing. . . 
'\u2264'
saving
'charmap' codec can't encode character '\u2264' in position 399: character maps to <undefined>
Found bad character, removing. . . 
'\u2264'
saving
'charmap' codec can't encode character '\u2264' in position 399: character maps to <undefined>
Found bad character, removing. . . 
'\u2264'
saving

I've tried a ton of variations:

data = data.replace(uchar, '')

data = data.replace(uchar.encode('utf-8').decode('utf-8'), '') etc. . .

I also tried u'\\2264', u'u\\2264'

I can't find this either in the dataframe. This returns nothing:

for col in data:
    if sum(data[col].astype(str).str.contains(u'\2264')) > 0:
        print(col)

Any help would be appreciated thanks!

Answer 1

您必须使用正则表达式替换功能： https : //pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.replace.html

df.replace(to_replace=r'^ba.$', value='new', regex=True)