我的 xpath 表达式有什么问题？

Question

I want to extract all the links in td whose class is u-ctitle.

import os
import urllib
import lxml.html
down='http://v.163.com/special/opencourse/bianchengdaolun.html'
file=urllib.urlopen(down).read()
root=lxml.html.document_fromstring(file)
namelist=root.xpath('//td[@class="u-ctitle"]/a')
len(namelist)

The output is [],there are so many td whose classis "u-ctitle" ,with firebug you ca get, why can't extract it?

My python version is 2.7.9.

It is no use to change file into other name.

Answer 1

Your XPath is correct. The problem is unrelated.

If you examine HTML, you will see following meta tag:

<meta http-equiv="Content-Type" content="text/html; charset=GBK" />

And in this code:

file=urllib.urlopen(down).read()
root=lxml.html.document_fromstring(file)

file is actually a bytes sequence, so decoding from GBK-encoded bytes to Unicode string is happening inside document_fromstring method.

The problem is, HTML encoding is not actually GBK and lxml decodes it incorrectly, leading to loss of data.

>>> file.decode('gbk')
Traceback (most recent call last):
  File "down.py", line 9, in <module>
    file.decode('gbk')
UnicodeDecodeError: 'gbk' codec can't decode bytes in position 7247-7248: illegal multibyte sequence

After some trial and error, we can find that actual encoding is GB_18030 . To make script work, you need to decode bytes manually:

root=lxml.html.document_fromstring(file.decode('GB18030'))