使用BeautifulSoup刮取产品名称

Question

I'm using BeautifulSoup (BS4) to build a scraper tool that will allow me to pull the product name from any TopShop.com product page, which sits between 'h1' tags. Can't figure out why the code I've written isn't working!

from urllib2 import urlopen
from bs4 import BeautifulSoup
import re

TopShop_URL = raw_input("Enter a TopShop Product URL")
ProductPage = urlopen(TopShop_URL).read()

soup = BeautifulSoup(ProductPage)

ProductNames = soup.find_all('h1')

print ProductNames

Answer 1

I get this working using requests ( http://docs.python-requests.org/en/latest/ )

from bs4 import BeautifulSoup
import requests

content = requests.get("TOPShop_URL").content
soup = BeautifulSoup(content)
product_names = soup.findAll("h1")
print product_names

Answer 2

Your code is correct, but the problem is that the div which includes the product name is dynamically generated via JavaScript. In order to be able to successfully parse this element you should mind using Selenium or a similar tool, that will allow you to parse the webpage after all the dom has been fully loaded.