Beautiful Soup - 选择没有类的下一个跨度元素的文本

Question

I am attempting to use Beautiful Soup to scrape movie quotes from rottentomatoes.com. The page source is interesting in that the quotes are directly proceeded by a span class "bold quote_actor", but the quote itself is in a span with no class, eg ( https://www.rottentomatoes.com/m/happy_gilmore/quotes/ ): screenshot of web source

I would like to use Beautiful Soup's find_all to capture all quotes, without the actor's name. I have tried many things with no success, such as:

 moviequotes = soup(input) for t in web_soup.findAll('span', {'class':'bold quote_actor'}): for item in t.parent.next_siblings: if isinstance(item, Tag): if 'class' in item.attrs and 'name' in item.attrs['class']: break print (item)

I would greatly appreciate any tips for how to navigate this code and to define the resulting plain text quotes into an object I use use with Pandas, etc.

Answer 1

I'm using CSS selectors to find the spans which contain quotes: div span + span . This finds any span element that is inside a div and has a direct sibling element of type span .

This way I also get the span s that contain actor names, so I filter them out by checking if they have a class or style attribute.

import bs4
import requests

url  = 'https://www.rottentomatoes.com/m/happy_gilmore/quotes/'
page = requests.get(url).text
soup = bs4.BeautifulSoup(page, 'lxml')

# CSS selector
selector = 'div span + span'

# find all the span elements which are a descendant of a div element
# and are a direct sibling of another span element 
quotes = soup.select(selector)

# now filter out the elements with actor names
data = []

for q in quotes:
    # only keep elements that don't have a class or style attribute
    if not (q.has_attr('class') or q.has_attr('style')):
        data.append(q)

for d in data:
    print(d.text)