Beautifullsoup 亞馬遜產品詳情

Question

我無法使用 requests 或 requests_html 抓取“產品詳細信息”部分（向下滾動網頁，您會找到它）html。 Find_all 返回 0 尺寸 object...有什么幫助嗎？

from requests import session
from requests_html import HTMLSession

s = HTMLSession()
#s = session()
r = s.get("https://www.amazon.com/dp/B094HWN66Y")
soup = BeautifulSoup(r.text, 'html.parser')
len(soup.find_all("div", {"id":"detailBulletsWrapper_feature_div"}))

Answer 1

具有不同信息的產品詳細信息：

代碼：

from bs4 import BeautifulSoup 
import requests

cookies = {'session': '131-1062572-6801905'}
headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'}

r = requests.get("https://www.amazon.com/dp/B094HWN66Y",headers=headers,cookies=cookies)
print(r)
soup = BeautifulSoup(r.text, 'lxml')
key = [x.get_text(strip=True).replace('\u200f\n','').replace('\u200e','').replace(':\n','').replace('\n', '').strip() for x in soup.select('ul.a-unordered-list.a-nostyle.a-vertical.a-spacing-none.detail-bullet-list > li > span > span.a-text-bold')][:13]
#print(key)

value = [x.get_text(strip=True) for x in soup.select('ul.a-unordered-list.a-nostyle.a-vertical.a-spacing-none.detail-bullet-list > li > span > span:nth-child(2)')]
#print(value)


product_details = {k:v for  k, v, in zip(key, value)}
print(product_details)

Output：

{'ASIN': 'B094HWN66Y', 'Publisher': 'Boldwood Books (September 7, 2021)', 'Publication date': 
'September 7, 2021', 'Language': 'English', 'File size': '1883 KB', 'Text-to-Speech': 'Enabled', 'Screen Reader': 'Supported', 'Enhanced typesetting': 'Enabled', 'X-Ray': 'Enabled', 'Word 
Wise': 'Enabled', 'Print length': '332 pages', 'Page numbers source ISBN': '1800487622', 'Lending': 'Not Enabled'}

Answer 2

這是一個如何使用bs4和requests抓取產品標題的示例，可以輕松擴展以從產品中獲取其他信息。

你的不起作用的原因是你的請求沒有標題，所以亞馬遜意識到你是一個機器人並且不希望你抓取他們的網站。 您的請求作為<Response [503]>返回並在r.text中進行了說明，這表明了這一點。

我相信亞馬遜為此有一個 API（他們可能希望您使用），但對於小規模的東西來說，像這樣抓取是沒問題的。

import requests
import bs4

# Amazon don't like you scrapeing them however these headers should stop them from noticing a small number of requests
HEADERS = ({'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64)AppleWebKit/537.36 (KHTML, like Gecko)Chrome/44.0.2403.157 Safari/537.36','Accept-Language': 'en-US, en;q=0.5'})


def main():
    url = "https://www.amazon.com/dp/B094HWN66Y"
    title = get_title(url)
    print("The title of %s is: %s" % (url, title))


def get_title(url: str) -> str:
    """Returns the title of the amazon product."""
    # The request
    r = requests.get(url, headers=HEADERS)

    # Parse the content
    soup = bs4.BeautifulSoup(r.content, 'html.parser')
    title = soup.find("span", attrs={"id": 'productTitle'}).string

    return title


if __name__ == "__main__":
    main()

Output： The title of https://www.amazon.com/dp/B094HWN66Y is: Will They, Won't They?