Python错误：“ utf8”编解码器无法解码位置85的字节0x92：无效的起始字节

Question

I am using python2.7 and lxml. My code is as below

import urllib
from lxml import html

def get_value(el):
    return get_text(el, 'value') or el.text_content()

response = urllib.urlopen('http://www.edmunds.com/dealerships/Texas/Frisco/DavidMcDavidHondaofFrisco/fullsales-504210667.html').read()
dom = html.fromstring(response)

try:
    description = get_value(dom.xpath("//div[@class='description item vcard']")[0].xpath(".//p[@class='sales-review-paragraph loose-spacing']")[0])
except IndexError, e:
    description = ''

The code crashes inside the try, giving an error

UnicodeDecodeError at /
'utf8' codec can't decode byte 0x92 in position 85: invalid start byte

The string that could not be encoded/decoded was: ouldn t be

I have tried using a lot of techniques including .encode('utf8'), but none does solve the problem. I have 2 question:

1. How to solve this problem
2. How can my app crash when the problem code is between a try except

Answer 1

The page is being served up with charset=ISO-8859-1 . Decode from that to unicode.

[

Answer 2

Your except clause only handles exceptions of the IndexError type. The problem was a UnicodeDecodeError, which is not an IndexError - so the exception is not handled by that except clause.

It's also not clear what 'get_value' does, and that may well be where the actual problem is arising.

Answer 3

1. • skip chars on Error, or decode it correctly to unicode.
2. • you only catch IndexError, not UnicodeDecodeError

Answer 4

1. decode the response to unicode, properly handling errors (ignore on error) before parsing with fromhtml.

2. catch the UnicodeDecodeError, or all errors.