从 Python 中的任何站点抓取链接标题的一般方法？

Question

Is there a "general" way to scrape link titles from any website in Python? For example, if I use the following code:

from urllib.request import url open
from bs4 import BeautifulSoup

site = "https://news.google.com"
html = urlopen(site)
soup = BeautifulSoup(html.read(), 'lxml');

titles = soup.findAll('span', attrs = { 'class' : 'titletext' }) 
for title in titles:
    print(title.contents)

I am able to extract nearly every headline title from news.google.com. However, if I use the same code at www.yahoo.com, I am unable to due to a different HTML formatting.

Is there a more general way to do this so that it works for most sites?

Answer 1

No, each site is different and if you make a more general scraper, it will get more data that isn't as specific as every headline title.

For instance the following would get every headline title from google and would also probably get them from yahoo also.

titles = soup.find_all('a') 
for title in titles:
    print(title.get_text())

However it would also get you all of the headers and other links which would muddy up your results. (there are approximately 150 links on that google page that aren't headlines)

Answer 2

Not, that's why we need CSS selector and XPath, but if there are small number of page, there is a convenient way to do that:

site = "https://news.google.com"
if 'google' in site:
    filters = {'name':'span', "class" : 'titletext' }
elif 'yahoo' in site:
    filters = {'name':'blala', "class" : 'blala' }
titles = soup.findAll(**filters) 
for title in titles:
    print(title.contents)