美丽的汤找不到Div

Question

I'm trying to get products for a project i'm working on from this page: Belk.com

I originally tried going very specific using

soup.find("ul", {"class" : "product_results"})

Nothing was happening, so I went very broad and just started searching all divs.

contentDiv = soup.find_all("div")
for div in contentDiv:
    print(div.get("class"))

When I do this I am getting only Div's for the top half of the page, which lead me to believe that there is an iframe that I wasn't getting into, but upon closer inspection I couldn't find the frame. Any thoughts on this?

Answer 1

This works for me

import httplib2
from bs4 import BeautifulSoup

http = httplib2.Http()
status, response = http.request('http://www.belk.com/AST/Main/Belk_Primary/Women/Shop/Accessories.jsp')

soup = BeautifulSoup(response)

res = soup.find('ul',{"class":"product_results"})
lis = res.findAll('li')
for j in lis:
    #your code
    pass