beautifulsoupobject.findAll() 返回一个空列表

Question

def ebay_scrape(urls):
    prices = []
    namets = []
    urlg = []
    for url in urls:
        try:
            res = requests.get(url)
            res.raise_for_status()
            soup = BeautifulSoup(res.text, 'html.parser')

        except (InvalidSchema, MissingSchema,):
            pass
        else: 
            try:
                price = soup.findAll (class_= "price").getText()
                prices.append(price)
                urlg.append(url)
            except (AttributeError):
                pass


    return urlg
urllist = (ebay_scrape(make_urls(names)))

I want to find extract all the prices from a query on an auction website. This script works fine when I do price = soup.find() but when I try soup.findAll it returns an empty list. I'm sure I'm just making some dumb mistake but any feedback would be appreciated!

Answer 1

I looked at the HTML for one of that auction site's pages and there's no element with a class attribute of "price" exactly.

Also, you can't call .getText() on a findAll() result.

You can use a regular expression instead to find anything containing the word price:

import re 

div_elements = soup.find_all('div',attrs={'class':re.compile('price')})

This turns up 252 results for one page.
Then you can collect the prices by doing this:

prices = []
for div_element in div_elements: 
    prices.append(div_element.text)

Result (which you'd have to clean up a bit):

In [191]: prices
Out[191]: 
['$49.34',
 '$92.00 | 46% off',
# ...