使用Python和BeautifulSoup编辑字符串并收集数据

Question

I'm trying to scrape the number of registrants for each division in the upcoming World Brazilian Jiu Jitsu Championships using Python and Beautiful Soup. (The goal in the end is to plot the number of competitors in each division, so the name of each registrant is immaterial.)

I'm able to pull some of the information I want using BeautifulSoup, but I can't seem to isolate exactly what I want.

import urllib2
from bs4 import BeautifulSoup
import fileinput
import matplotlib

page = urllib2.urlopen("https://www.ibjjfdb.com/ChampionshipResults/926/PublicRegistrations?lang=en-US").read()
soup = BeautifulSoup(page,'html.parser')

My original plan was to go through all the data, count the names of the people registered in each division, save it as a data set and then plot the information by weight, age, etc. I now see that there's a counter at the bottom of each division, so all I really need to do is identify the weight division and the total.

Originally I tried using soup.find_all() to pull the information I wanted, but it's giving me a lot of excess. I started with trying to pull the whole table using a command I pulled from another StackOverflow question:

result = soup.find_all(lambda tag: tag.name == 'div' and tag.get('class') == ["row"])

which gives one long string containing what I want:

[<div class="row">\n<h4>BLUE / Juvenile 1 / Male / Rooster</h4>\n<table class="table table-striped">\n<tr>\n<td style="width:40%;">\r\n                                    Atos Jiu-Jitsu\r\n [...]

To narrow it down, I searched for the heading that just gives the weight classes:

all_divisions = [division for division in soup.find_all("h4")]

which works well and gives:

[<h4>BLUE / Juvenile 1 / Male / Rooster</h4>, <h4>BLUE / Juvenile 1 / Male / Light-Feather</h4>, <h4>BLUE / Juvenile 1 / Male / Feather</h4>, <h4>BLUE / Juvenile 1 / Male / Light</h4>, [...]

I want to be able divide these into their separate groups so I can plot it. How can I split this up? I tried using all_divisions[0].split() but apparently soup.find_all() returns NoneType objects instead of strings.

Next, I'm trying to get the total. I can isolate the string "Total:" using soup.find_all('strong') , but it can't seem to get the actual number since that's not included in the bold:

<td colspan="2">
  <strong>Total:</strong> 8
</td>

I tried to go out one and pick up that row of the table using soup.find_all(lambda tag: tag.name == 'td' and tag.get('colspan') == ["2"]) but it doesn't return anything when I do that. How do I pick up the total number of competitors?

(Since the number of divisions and totals are listed together and in order, I don't think I need to worry about matching the total to the division as long as I'm careful about the ordering.)

Answer 1

You can use a loop to get the values you need.

#Segregate data by rows
    rows = soup.findAll('div',{'class':'row'})

    #Create empty lists
    divisions = []
    competitors = []

    #Create and append respective lists with a loop
    for row in rows:
        raw_division = row.h4.text
        division = raw_division.split('/')
        divisions.append(division)

        raw_competitors = row.tfoot.tr.td.text.strip()
        competitor = raw_competitors.replace('Total: ','')
        competitors.append(competitor)

You should now have a list of lists --> divisions as well as a list --> competitors.