I am currently scraping a website and need to obtain certain font-sizes, such that in style="font-size: 140%", I want to grab 140% or preferably just 140 so that I can use it in certain calculations since each one will have a different font-size.
More specifically, I want to grab the font-sizes from tags like this...
<div style="font-size: 141%; line-height: 110%"><a href="engenremap-latinrock.html" style="color: #AF7E1C">latin rock</a></div> <div style="font-size: 139%; line-height: 110%"><a href="engenremap-mexicanindie.html" style="color: #B18230">mexican indie</a></div>
Typically I'd be able to do this with no problem. However, the issue I am having is that there are no distinguishing tags on what I want to grab and there are a bunch of preceding lines consisting of tags with font-size like this...
<tr valign=top class="datarow firstrow" style="white-space: nowrap"><td align=right class=note style="font-size: 20px; line-height: 24px">1</td><td style="font-size: 20px; line-height: 24px"> <a href="spotify:playlist:0DsV1U8e3xXsmsDSaW88XT" class=note target=spotify title="See this playlist in Spotify.">☊</a></td><td class=note style="font-size: 20px; line-height: 24px"><a href="?scope=MX&vector=activity" title="Show only schools from Mexico." style="color: #BA890D">Mexico</a></td><td class=note style="font-size: 20px; line-height: 24px"><a href="?root=Universidad%20Nacional%20Aut%C3%B3noma%20De%20M%C3%A9xico%20%28UNAM%29&scope=all" title="Re-sort the list by similarity to Universidad Nacional Autónoma De México (UNAM)." style="color: #BA890D">Universidad Nacional Autónoma De México (UNAM)</a></td></tr> <tr valign=top class="datarow " style="white-space: nowrap"><td align=right class=note style="font-size: 20px; line-height: 24px">2</td><td style="font-size: 20px; line-height: 24px"> <a href="spotify:playlist:5QAomgXhxwYjg975DWtTTv" class=note target=spotify title="See this playlist in Spotify.">☊</a></td><td class=note style="font-size: 20px; line-height: 24px"><a href="?scope=US&vector=activity" title="Show only schools from USA." style="color: #948F04">USA</a></td><td class=note style="font-size: 20px; line-height: 24px"><a href="?root=Texas%20A%20%26%20M%20University-College%20Station&scope=all" title="Re-sort the list by similarity to Texas A & M University-College Station." style="color: #948F04">Texas A & M University-College Station</a></td></tr>
Keeping in mind that I already traverse the links in the previous snippet (they are static and remain in the same position as I traverse) and that the tags from the first snippet (genres at schools) change for every link (schools), how would I go about ignoring the font-sizes from the tr tags and only grab the font-sizes from the first HTML snippet? I'm sure this has a simple solution, but I'd appreciate any help.
** I am already traversing through the links and grabbing the respective genres for each school, I just need to also grab the font-sizes for those specific genres as well. **
Here is some of my code to provide more context...
data = [] # used to sort between country and university <td> tags
links = [] # stores links from clicking on the university name and used to get genres
countries = [] #
universities = [] # indices match for these lists
spotifyLinks = [] #
fontSizes = [] #
genres = [[]] #
genres_weight = [[]] #
page = requests.get("http://everynoise.com/everyschool.cgi") # stores response from Every Noise into page
soup = BeautifulSoup(page.content, 'html.parser') # used to create data list
soup1 = BeautifulSoup(page.content, 'html.parser') # used to create links and spotifyLinks lists
soupList = list(soup.find_all('td', class_="note")) # creates list of <td> tags where class="note"
for soup in soupList: #
if not soup.get_text().isnumeric(): # stores all country and university names in data list
data.append(soup.get_text()) #
for i in range(len(data)): #
if i%2 == 0: # separates data list into two individual lists
countries.append(data[i]) # for country and university names respectively
else: #
universities.append(data[i]) #
for a in soup1.find_all('a', attrs={'href': re.compile("\?root=")}):
links.append('http://everynoise.com/everyschool.cgi' + a['href'])
for a in soup1.find_all('a', attrs={'href': re.compile("spotify:playlist:")}):
spotifyLinks.append('https://open.spotify.com/playlist/' + a['href'][17:])
spotifyLinks = spotifyLinks[:-1]
linkSubset = links[0:4] # subset of links for quicker testing
j=1
for link in linkSubset: # switch out linkSubset with links for full dataset
time.sleep(1) # so we don't spam their servers
schoolGenres = []
nextPage = urllib.urlopen(url=link)
bs_obj = BeautifulSoup(nextPage, "html.parser")
for a in bs_obj.find_all('a', attrs={'href': re.compile("^engenremap-")}):
schoolGenres.append(a.get_text())
genres.append(schoolGenres)
print "Scraping...", j
j=j+1
genres = genres[1:]
distinct_genres = set()
for genre in genres:
distinct_genres.update(genre)
print "\nDistinct Genres:", distinct_genres
Edit/Answer: This ended up being solved by using a slightly modified version of the answer selected.
pattern = re.compile(r'font-size: (\d+)')
for a in bs_obj.select('div[style*="font-size"]'):
genreWeights.append(int(pattern.search(str(a)).group(1)))
You can search for a tag that contains certain text, and than extract the value of font-size
. For example:
import re
from bs4 import BeautifulSoup
txt = """<div style="font-size: 141%; line-height: 110%"><a href="engenremap-latinrock.html" style="color: #AF7E1C">latin rock</a></div>
<div style="font-size: 139%; line-height: 110%"><a href="engenremap-mexicanindie.html" style="color: #B18230">mexican indie</a></div>
<div style="font-size: 139%; line-height: 110%"><a href="engenremap-mexicanindie.html" style="color: #B18230">hello world/a></div>"""
soup = BeautifulSoup(txt, "html.parser")
pattern = re.compile(r'font-size: (\d+)')
for tag in soup.select("div:contains(latin, mexican)"):
font_size = pattern.search(str(tag)).group(1)
print(font_size)
Output:
141
139
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.