即使带有标头也发送请求时出现 Python 403 禁止错误

Question

正如上面的标题所述，我收到 403 错误。 生成的 URL 是有效的，我可以打印它们，然后在我的浏览器中打开它们就好了。 我得到了整个请求标头，但仍然是 403 Forbidden，有人可以帮我解决吗？

    import requests
    from bs4 import BeautifulSoup
    header = {
        "sec-ch-ua": '" Not A;Brand";v="99", "Chromium";v="99", "Microsoft Edge";v="99"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "Windows",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.30"
    }
    url="https://www.nadirkitap.com/"
    get = requests.get(url,headers=header)
    
    print(get.status_code)

Answer 1

查看响应文本 - 它告诉网站受 cloudflare 保护，您需要激活 JavaScript。 因为requests不处理这个，你可以改用selenium 。

例子

包含基于driver.page_source的BeautifulSoup对象并打印书名列表：

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

service = Service('PATH TO YOUR CHROMEDRIVER')
driver = webdriver.Chrome(service=service)

driver.get('https://www.nadirkitap.com/')
soup = BeautifulSoup(driver.page_source)

print([t['title'] for t in soup.select('a[title]')])

输出

['İkinci el kitap, yeni kitap, dergi, efemera', 'İkinci el kitap, yeni kitap, dergi, efemera', 'İkinci el kitap ve yeni kitap', 'Bilim ve Teknik Kitapları', 'Çizgi Roman Kitapları', 'Çocuk Kitapları', 'Dini Kitaplar', 'Edebiyat Kitapları', 'Ekonomi ve İş Dünyası Kitapları', 'Felsefe Kitapları', 'Hukuk Kitapları', 'Osmanlıca Kitaplar',...]