用美丽的汤提取可变元素

Question

I'm trying to scrape the rating of yelp restaurants, with no luck. I'm using beautiful soup to do so

Basically, the source code looks like this:

<div class="i-stars i-stars--regular-5 rating-large" title="5.0 star rating">
    <img class="offscreen" height="303" src="https://s3-media2.fl.yelpcdn.com/assets/srv0/yelp_design_web/9b34e39ccbeb/assets/img/stars/stars.png" width="84" alt="5.0 star rating">
</div>

As you can see, the class name changes given the rating, so I'm trying to have somekind of “partial” match with my find function

rating = r.find_all('div', {'class':'i-stars i-stars--regular'}).get('title', 'No title attribute')
print(rating)

But it does not seem to work.

Answer 1

Use regex -

import re
rating = [x.get('title', 'No title attribute') for x in r.find_all('div', attrs={"class": re.compile("i-stars i-stars--regular-")})]
print(rating)

Answer 2

if you need partial search for class use.

s = """<div class="i-stars i-stars--regular-5 rating-large" title="5.0 star rating">
    <img class="offscreen" height="303" src="https://s3-media2.fl.yelpcdn.com/assets/srv0/yelp_design_web/9b34e39ccbeb/assets/img/stars/stars.png" width="84" alt="5.0 star rating">
</div>"""

from bs4 import BeautifulSoup
r = BeautifulSoup(s, "html.parser")
rating = r.find_all('div')
for i in rating:
    if "i-stars i-stars--regular-5" in " ".join(i["class"]):
        print(i.get('title', 'No title attribute')) 

Output:

5.0 star rating

Answer 3

You can do it by using a function :

from bs4 import BeautifulSoup
import re
html_text = '<div class="i-stars i-stars--regular-5 rating-large" title="5.0 star rating">\n    <img class="offscreen" height="303" src="https://s3-media2.fl.yelpcdn.com/assets/srv0/yelp_design_web/9b34e39ccbeb/assets/img/stars/stars.png" width="84" alt="5.0 star rating">\n</div'
soup = BeautifulSoup(html_text, 'html.parser')
soup.find_all(class_ = lambda x:re.search(r"i\-stars i\-stars\-\-regular\-\d rating\-\w+",x))

give out a regex pattern within it based on your requirement. Here I've marked it for any rating size and star.

Answer 4

find_all() returns a list of all the matches. So, you can use .get() on a list. To get only one item, you'll have to use find_all(...)[0] , or, even better, the find() function; which returns the first match.

Also, since the class name changes according to the rating, you can use the classes which are constant and add them in the list. For example, here, i-starts and rating-large seem to be constant. So, you can use this:

html = '''
<div class="i-stars i-stars--regular-5 rating-large" title="5.0 star rating">
    <img class="offscreen" height="303" src="https://s3-media2.fl.yelpcdn.com/assets/srv0/yelp_design_web/9b34e39ccbeb/assets/img/stars/stars.png" width="84" alt="5.0 star rating">
</div>'''
soup = BeautifulSoup(html, 'lxml')
rating = soup.find('div', {'class': ['i-stars', 'rating-large']}).get('title', 'No title attribute')
print(rating)
# 5.0 star rating

But, you have to be careful while using the classes in a list, as it will also match the classes that have only one of the classes, like class="i-stars" . If such cases exist, you can use the following selector:

ratings = soup.select_one('div.i-stars.rating-large').get('title', 'No title attribute')