如何在不转换为utf-8的情况下处理无效的unicode BeautifulSoup？

Question

So I am trying to parse a website's HTML with python and there is one troublesome character u011f that gives the following error:

Function call:   soup = BeautifulSoup(response, "html.parser")
                 print (soup)

Error: UnicodeEncodeError: 'charmap' codec can't encode character '\u011f'

If I do this instead and encode to utf-8,

soup = BeautifulSoup(response, "html.parser").encode('utf-8') 

It removes the error, but I cannot do that because I am calling the find function later, and it must be in unicode. If I call the find function after encoding to utf-8, I receive the following error:

Function call:   worksTable = soup.find('tbody', attrs={'id': 'some_id'})
Error: TypeError: find() takes no keyword arguments

I have spent hours already on this code and could not find any answers here that fit my case. Any help would be appreciated.

Answer 1

When you encode the soup, it becomes a string .

The following line of code

BeautifulSoup(response, "html.parser").encode('utf-8')

will return a string object and so will not support a find(tagname, attrs={}) method call which should be used on a BeautifulSoup object.

I think you should encode the response text before making the soup to get a better result.

responseTxt = response.text.encode('UTF-8')
soup = BeautifulSoup(responseTxt, 'html.parser')
idv = soup.find('tbody', attrs={'id': 'some_id'})
print(idv.text)

Answer 2

So I found out that it was problem with my desktop codec. The same code runs fine on my laptop. I'm very confused about this, but will find a way to manage.

Answer 3

您可以尝试在find()而不是汤上使用encode() ，这是一个示例：

worksTable = soup.find('tbody', attrs={'id': 'some_id'}).text.encode('utf-8')