使用beautifulsoup python在span类HTML中刮取值

Question

I am trying to scrape data inside a span class and putting that data inside a DataFrame using Beautifulsoup. So far, I've been successful at getting to the right place of the webpage. But can't seem to be able to scrape the keywords and numbers next to "Happiness", "Sadness".

<span class="text-border tooltips" data-original-title="Happiness 84%
 Sadness 80%
 " data-placement="left" data-toggle="tooltip">More stats</span>,
 <span class="text-border tooltips" data-original-title="Happiness 70%
 Sadness 59%
 " data-placement="left" data-toggle="tooltip">More stats</span>

Would be super helpful if someone could help me figure out to scrape all numbers next to Happiness and Sadness, and have them as columns in a pandas DataFrame.

Thanks a lot

Answer 1

You could do something like

from bs4 import BeautifulSoup

s = """
<span class="text-border tooltips" data-original-title="Happiness 84%
 Sadness 80%
 " data-placement="left" data-toggle="tooltip">More stats</span>,
 <span class="text-border tooltips" data-original-title="Happiness 70%
 Sadness 59%
 " data-placement="left" data-toggle="tooltip">More stats</span>
"""

soup = BeautifulSoup(s, "lxml")
spans = soup.find_all("span") #get all spans
for span in spans:
    data = span["data-original-title"].split("\n") #get attr and split by \n 
    happiness = data[0][:-1].replace("Happiness ", "") #remove % and remove words
    sadness = data[1][:-1].replace("Sadness ", "")
    print("{} {}".format(happiness, sadness))

Answer 2

If it's guaranteed that all the spans will have a data-original-title... and If the title will always in in the format of "Happiness<SPACE><PERCENTAGE><NEW LINE>Sadness<SPACE><PERCENTAGE>" Then below should work out for you.

>>> import itertools
>>> import re
>>> import pandas as pd
>>> import bs4
>>> html = """<span class="text-border tooltips" data-original-title="Happiness 84%
...  Sadness 80%
...  " data-placement="left" data-toggle="tooltip">More stats</span>,
...  <span class="text-border tooltips" data-original-title="Happiness 70%
...  Sadness 59%
...  " data-placement="left" data-toggle="tooltip">More stats</span>"""
>>> soup = bs4.BeautifulSoup(html, 'lxml')
>>> all_rows = []
>>> for span in soup.find_all('span'):
...     title_eles = re.split(' |\n', span['data-original-title'])
...     title_eles = list(filter(None, title_eles))
...     row = dict(itertools.zip_longest(title_eles[::2], title_eles[1::2], fillvalue=""))
...     all_rows.append(row)
...
>>> pd.DataFrame(all_rows)
  Happiness Sadness
0       84%     80%
1       70%     59%

Also the reason why soup.find_all(class_='data-original-title') returns empty is because data-original-title is an attribute in your HTML. It's not a class.