Scrapy选择器可能由于Java语言而无法返回所需的字符

Question

I'm trying to scrape data from this Chinese webpage http://bxt.harbin.gov.cn/hrb_bzbxt/disshow.php?id=551950 .

In Scrapy shell, I cannot get any text in any td elements. For example, response.xpath("/html/body/center[2]/table/tbody/tr[2]/td[3]/text()").extract() returns an empty list. The same thing is returned for other similar commands too. When I inspect the html more closely, I find this in the head element: "script language = "javascript". I'm not sure if this is the cause of the problem. Could anybody help me figure out? I searched Stackoverflow for related topics, but it's too complex for me to grasp. Thank you for your help!

Answer 1

the problem here is that you are using a full path to get to the information you want, this isn't necessary, so no need to follow html -> body -> center , etc.

You could just go directly to the td information you need, with something like:

response.xpath('//td/text()')

which will return a list of selectors (every text inside a td tag) to iterate with the information you need.