如何从字符串中删除某些utf-8字符？

Question

In my case I want to remove specifically the „ and the ” characters from a string. I use BeautifulSoup to parse certain html paragraphs, and get a substring from them. So far my code looks like this:

# -*- coding: cp1252 -*-
from bs4 import BeautifulSoup as bs
import re

soup = bs(open("file.xhtml"), 'html.parser')

for tag in soup.find_all('p', {"class": "fnp2"}) :
    line = unicode(str(tag).split(':')[0], "utf-8")
    line = re.sub('(<p class="fnp2">)(\d+) ', '', line)
    line = line.replace('„', '')
    print line

But for that, I always receive a UnicodeDecodeError :

line = line.replace('„', '')

UnicodeDecodeError: 'ascii' codec can't decode byte 0x84 in position
0: ordinal not in range(128)

What would be a solution for this?

Answer 1

The line variable in your code is a unicode object. When you call line.replace Python expects the first argument to also be a unicode object. If you provide a str object instead, Python will try to automatically decode it into a unicode string using the system default encoding (which you can check via sys.getdefaultencoding() ).

Apparently, the system encoding is ascii in your case. The byte string '„' cannot be decoded using the ascii codec, because '„' is not an ACII symbol, which causes the Exception that you see.

You could fix the problem by changing the default system encoding to the same one you used to provide the '„' string (CP1252, I guess), however such a fix is only interesting from the academic point of view, as it just sweeps the issue under the carpet.

A proper, safe and easy fix to your problem would be to simply provide a unicode object to the replace method in the first place. This would be as simple as replacing '„' with u'„' in your code.