無法使用請求模塊從 static 網頁中抓取不同的管道工名稱

Question

在過去的幾個小時里，我一直在嘗試使用 requests 模塊從該網頁上抓取不同的plumber names ，因為該網站的內容是 static 並且在頁面源代碼中也可用（Ctrl + U）。

但是，當我運行腳本時，出現以下錯誤：

raise TooManyRedirects('Exceeded {} redirects.'.format(self.max_redirects), response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

這就是我正在嘗試的方式：

from bs4 import BeautifulSoup
import requests
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

link = 'https://www.yellowpages.co.za/search?what=plumber&where=bryanston+west%2c+sandton%2c+gauteng&pg=2'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    'Referer': 'https://www.yellowpages.co.za/search?what=plumber&where=bryanston+west%2c+sandton%2c+gauteng&pg=1',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9',
    'Host': 'www.yellowpages.co.za',
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link,verify=False)
    soup = BeautifulSoup(res.text,"html.parser")
    for shop_name in soup.select("h5.nameOverflow"):
        print(shop_name.get_text(strip=True))

Answer 1

我試圖讓它與requests一起工作，但最后我使用了標准 python 的urlopen() ：

import ssl
from bs4 import BeautifulSoup
from urllib.request import urlopen


ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

response = urlopen(
    "https://www.yellowpages.co.za/search?what=plumber&where=bryanston+west%2c+sandton%2c+gauteng&pg=2",
    context=ctx,
)

soup = BeautifulSoup(response, "html.parser")
for shop_name in soup.select("h5.nameOverflow"):
    print(shop_name.get_text(strip=True))

印刷：

Absolutely Fast Plumbing Co CC
Property Matters Gauteng (Pty) Ltd
Plumlite
Geyser Man
Daryn's Plumbing Services (Pty) Ltd
Electroc
Outek Engineers CC
Bryanston Plumbing (Pty) Ltd
Mage Plumbing & Electrical
Fourways Plumbing
Matrix Plumber
Angel Plumbers
Renovations And Maintenance Services
DCB Supplies
Call Us Plumbing
A B A Group
Action Plumbing
Clearline Plumbing Services
Capital Plumbing Supplies
AGD Plumbing

Answer 2

您收到此錯誤的原因是，當您通過提供的鏈接訪問該網站時，它被重定向了幾次。 看看這個線程，讓我知道它是否適合你