使用BeautifulSoup查找與特定關鍵字相關的鏈接

Question

我必須修改此代碼，以便抓取僅保留包含特定關鍵字的鏈接。 以我為例，我是在刮報紙頁上查找與“英國脫歐”一詞有關的新聞。

我試過修改parse_links方法，使其僅保留其中包含“ Brexit”的鏈接（或“ a”標簽），但似乎不起作用。

我應該在哪里放置條件？

import requests
from bs4 import BeautifulSoup
from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urljoin, urlparse

class MultiThreadScraper:

    def __init__(self, base_url):

        self.base_url = base_url
        self.root_url = '{}://{}'.format(urlparse(self.base_url).scheme, urlparse(self.base_url).netloc)
        self.pool = ThreadPoolExecutor(max_workers=20)
        self.scraped_pages = set([])
        self.to_crawl = Queue(10)
        self.to_crawl.put(self.base_url)

    def parse_links(self, html):
        soup = BeautifulSoup(html, 'html.parser')
        links = soup.find_all('a', href=True)
        for link in links:
            url = link['href']
            if url.startswith('/') or url.startswith(self.root_url):
                url = urljoin(self.root_url, url)
                if url not in self.scraped_pages:
                    self.to_crawl.put(url)

    def scrape_info(self, html):
        return

    def post_scrape_callback(self, res):
        result = res.result()
        if result and result.status_code == 200:
            self.parse_links(result.text)
            self.scrape_info(result.text)

    def scrape_page(self, url):
        try:
            res = requests.get(url, timeout=(3, 30))
            return res
        except requests.RequestException:
            return

    def run_scraper(self):
        while True:
            try:
                target_url = self.to_crawl.get(timeout=60)
                if target_url not in self.scraped_pages:
                    print("Scraping URL: {}".format(target_url))
                    self.scraped_pages.add(target_url)
                    job = self.pool.submit(self.scrape_page, target_url)
                    job.add_done_callback(self.post_scrape_callback)
            except Empty:
                return
            except Exception as e:
                print(e)
                continue
if __name__ == '__main__':
    s = MultiThreadScraper("https://elpais.com/")
    s.run_scraper()

Answer 1

您可以使用getText()方法獲取元素的文本，並檢查字符串是否實際包含“ Brexit”：

if "Brexit" in link.getText().split():
     url = link["href"]

Answer 2

您需要導入re模塊以獲得特定的文本值。嘗試以下代碼。

import re
 links = soup.find_all('a', text=re.compile("Brexit"))

這應返回僅包含英國退歐的鏈接。

Answer 3

我在此功能中添加了一個檢查。 看看這是否對您有幫助：

def parse_links(self, html):
    soup = BeautifulSoup(html, 'html.parser')
    links = soup.find_all('a', href=True)
    for link in links:
        if 'BREXIT' in link.text.upper():  #<------ new if statement
            url = link['href']
            if url.startswith('/') or url.startswith(self.root_url):
                url = urljoin(self.root_url, url)
                if url not in self.scraped_pages:
                    self.to_crawl.put(url)