美丽的汤解析亚马逊页面

Question

Hi I'm trying to parse Amazon's page for book details so I'm using beautiful soup

link: https://www.amazon.com/Dogs-Purpose-Novel-Humans/dp/0765326264/ref=sr_1_1?s=electronics&ie=UTF8&qid=1489776209&sr=1-1&keywords=books

from bs4 import BeautifulSoup
import requests

url = raw_input("Enter a website to extract the URL's from: ")
r  = requests.get(url)

data = r.text

soup = BeautifulSoup(data, "lxml")

#Grab book details
print soup.find("table", {"id": "productDetailsTable" })

But when I try this code I get None as a result, I'm sure the id productDetailsTable exist, and when I try running this code with dummy html it works just not with a url?

Answer 1

I did not see productDetailsTable on https://www.amazon.com

I had to do https://www.amazon.com/ in order to receive the html data.

Here is my slightly modified Python 3 code.

from bs4 import BeautifulSoup
import requests

url = input("Enter a website to extract the URL's from: ")
r  = requests.get(url)

data = r.text

soup = BeautifulSoup(data, "lxml")

print(soup.text)

It prints the html for the page.

You'll notice that amazon is smart. The html includes the Robot Check:

if (true === true) {
var ue_t0 = (+ new Date()),
    ue_csm = window,
    ue = { t0: ue_t0, d: function() { return (+new Date() - ue_t0); } },
    ue_furl = "fls-na.amazon.com",
    ue_mid = "ATVPDKIKX0DER",
    ue_sid = (document.cookie.match(/session-id=([0-9-]+)/) || [])[1],
    ue_sn = "opfcaptcha.amazon.com",
    ue_id = 'R8D7EEN5FVS7RWC2M549';
}
Enter the characters you see below
Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.

It is keeping you from reading Amazon's page. You'll have to do more, probably with requests and include headers and cookie information.