如何使用 XPath 在 Python 的 2 层内抓取网站

Question

So I am trying to crawl this website and an example url is https://www.rfa.org/cantonese/news/SARS-12312019075620.html

I am just trying to get the text, however, you can see that some of the text is under p tag and some of them are just between br . I do not want to get text description of the pictures so I cannot crawl everything.

This is what I have so far that only gets text under p

//*//div[@id="storytext"]/p/text()

But how can I get every text but not description of pictures and other unnecessary information.

So there are 2 layers. The first is p and the other one is text between br . The description of the pictures are always in 3 layers.

Answer 1

Assuming you're using LXML. You should write a specific XPath (using axes):

from lxml import html
import requests
page = requests.get('https://www.rfa.org/cantonese/news/SARS-12312019075620.html')
tree = html.fromstring(page.content)
news = tree.xpath('//div[@id="storytext"]//text()[normalize-space()][parent::div[@id="storytext"] or ancestor::p]')
print (news)

Output:

Use regex to make final cleaning.

Urls used to test the XPath (it should work for all types of pages: news, features, talkshow, etc.):

• https://www.rfa.org/cantonese/news/SARS-12312019075620.html
• https://www.rfa.org/cantonese/news/htm/tw-han-06062020085146.html
• https://www.rfa.org/cantonese/news/htm/hk-civil-servant-06072020111622.html
• https://www.rfa.org/cantonese/features/hottopic/hk2020-2-05282020160237.html
• https://www.rfa.org/cantonese/talkshows/tptk/tptk-06032020085708.html