Python 美汤 find_all()

Question

I am trying to use find_all() on the below html;

http://www.simon.com/mall

Based on advice on other threads, I ran the link through the below site and it found errors, but I am not sure how the errors shown may be hurting what I am trying to do in Beautiful Soup.

https://validator.w3.org/

Here is my code;

from requests import get

url = 'http://www.simon.com/mall'
response = get(url)

from bs4 import BeautifulSoup

html = BeautifulSoup(response.text, 'html5lib')
mall_list = html.find_all('div', class_ = 'col-xl-4 col-md-6 ')

print(type(mall_list))
print(len(mall_list))

The result is;

"C:\Program Files\Anaconda3\python.exe" C:/Users/Chris/PycharmProjects/IT485/src/GetMalls.py
<class 'bs4.element.ResultSet'>
0

Process finished with exit code 0

I know there are hundreds of these divs in the HTML. Why am I not getting any matches?

Answer 1

I sometime use BeautifulSoup too. The problem lies in the way you get the attributes. The full working code can be seen bellow:

import requests
from bs4 import BeautifulSoup

url = 'http://www.simon.com/mall'
response = requests.get(url)
html = BeautifulSoup(response.text)
mall_list = html.find_all('div', attrs={'class': 'col-lg-4 col-md-6'})[1].find_all('option')
malls = []

for mall in mall_list:
    if mall.get('value') == '':
        continue
    malls.append(mall.text)

print(malls)
print(type(malls))
print(len(malls))

Answer 2

Your code looks fine, however, when I visit the simon.com/mall link and check Chrome Dev Tools there doesn't seem to be any instances of the class 'col-xl-4 col-md-6 '.

Try testing your code with 'col-xl-2' and you should see some results.

Answer 3

Assuming that you are trying to parse the title and location of different products from that page (mentioned in your script). The thing is the content of that page are generated dynamically so you can't catch it with requests; rather, you need to use any browser simulator like selenium that is What i did in my below code. Give this a try:

from selenium import webdriver
from bs4 import BeautifulSoup
import time

driver = webdriver.Chrome()
driver.get('http://www.simon.com/mall')
time.sleep(3)

soup = BeautifulSoup(driver.page_source, 'lxml')
driver.quit()

for item in soup.find_all(class_="mall-list-item-text"):
    name = item.find_all(class_='mall-list-item-name')[0].text
    location = item.find_all(class_='mall-list-item-location')[0].text
    print(name,location)

Results:

ABQ Uptown Albuquerque, NM
Albertville Premium Outlets® Albertville, MN
Allen Premium Outlets® Allen, TX
Anchorage 5th Avenue Mall Anchorage, AK
Apple Blossom Mall Winchester, VA