美丽的汤findall返回不同的结果

Question

I am trying to parse through a div class from an html table on Amazon, and when I run the code, find_all() sometimes returns the right div classes that I am looking for, and other times it will return an empty list. Any ideas on why the results vary?

I am pulling from this url: https://www.amazon.com/dp/B0767653BK

My code:

req = requests.get('https://www.amazon.com/dp/B0767653BK')
page = req.text
BSoup = BeautifulSoup(page, 'html.parser')
divClass = Bsoup.find_all('div', class_='a-section a-spacing-none a-padding-none overflow_ellipsis')

Answer 1

It is better to use a beautifulsoup selector when trying to find all elements with a combination of CSS classes:

from bs4 import BeautifulSoup
import requests

req = requests.get('https://www.amazon.com/dp/B0767653BK')
soup = BeautifulSoup(req.text, 'html.parser')

for div_class in soup.select('div.a-section.a-spacing-none.a-padding-none.overflow_ellipsis'):
    print div_class.get_text(strip=True)

This is preferable as it allows the four class elements to be present in any order. So if the page decides to change the ordering of the classes, it will still find them.

Take a look at Searching by CSS class in the documenation.