使用Python和BeautifulSoup刮刮Amazon数据时出错

Question

I just started with Python and I have this strange behaviour that Python gives me an Error most of the time and sometimes it compiles my code correctly.

import requests
from bs4 import BeautifulSoup

jblCharge4URL = 'https://www.amazon.de/JBL-Charge-Bluetooth-Lautsprecher-Schwarz-integrierter/dp/B07HGHRYCY/ref=sr_1_2_sspa?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&keywords=jbl+charge+4&qid=1562775856&s=gateway&sr=8-2-spons&psc=1'

def get_page(url):
    page = requests.get(url, headers=headers)
    soup = BeautifulSoup(page.content, 'html.parser')
    return soup

def get_product_name(url):
    soup = get_page(url)
    try:
        title = soup.find(id="productTitle").get_text()
        print("SUCCESS")
    except AttributeError:
        print("ERROR")
while(True)
    print(get_product_name(jblCharge4URL))

Console Output:

ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
**SUCCESS**  
None  
ERROR  
None  
**SUCCESS**  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None  
ERROR  
None

Thanks in Advance

Answer 1

I made a few adjustments to your code and this should get you back on the right track:

import requests
from bs4 import BeautifulSoup

jblCharge4URL = 'https://www.amazon.de/JBL-Charge-Bluetooth-Lautsprecher-Schwarz-      integrierter/dp/B07HGHRYCY/ref=sr_1_2_sspa?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&  keywords=jbl+charge+4&qid=1562775856&s=gateway&sr=8-2-spons&psc=1'

def get_page(url):
    page = requests.get(url)
    soup = BeautifulSoup(page.text, 'html.parser')
    return soup

def get_product_name(url):
    soup = get_page(url)
    try:
        title = soup.find(id="productTitle")
        print("SUCCESS")

    except AttributeError:
        print("ERROR")
    return(title)   
print(get_product_name(jblCharge4URL))

Answer 2

What headers are you using in page = requests.get(url, headers=headers) ? You would want something that tricks the server into believing that you are a genuine user and not a script. I would recommend using something basic like

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'}

Also, you might want to print the value of variable soup in your exception as you debug this issue. Printing soup will give you the HTML of the page and you can then dig inside the source code to understand where the issue lies.

Answer 3

Apart from using requests and BeautifulSoup combination, you could also use the requests-html package to download your web page and parse the content at the same time. An example of using requests-html would be:

from requests_html import HTMLSession

url = r"https://www.amazon.de/JBL-Charge-Bluetooth-Lautsprecher-Schwarz-integrierter/dp/B07HGHRYCY/"

req = HTMLSession().get(url)
product_title = req.html.find("#productTitle", first=True)
print(product_title.text)  #JBL Charge 4 Bluetooth-Lautsprecher in Schwarz – Wasserfeste, portable Boombox mit integrierter Powerbank – Mit nur einer Akku-Ladung bis zu 20 Stunden kabellos Musik streamen

Hope it helps.