餐桌和美丽汤的问题

Question

I'm trying to tags that are nested in a tr tag, but the identifier that I'm using to find the correct value is nested in another td within the tr tag.

That is, I'm using the website LoLKing

And trying to scrape it for statistics based on a name, for example, Ahri.

The HTML is:

<tr>
            <td data-sorttype="string" data-sortval="Ahri" style="text-align: left;">
                <div style="display: table-cell;">
                <div class="champion-list-icon" style="background:url(//lkimg.zamimg.com/shared/riot/images/champions/103_32.png)">
                    <a style="display: inline-block; width: 28px; height: 28px;" href="/champions/ahri"></a>
                </div>
                </div>
                <div style="display: table-cell; vertical-align: middle; padding-top: 3px; padding-left: 5px;"><a href="/champions/ahri">Ahri</a></div>
            </td>
            <td style="text-align: center;"  data-sortval="975"><img src='//lkimg.zamimg.com/images/rp_logo.png' width='18' class='champion-price-icon'>975</td>
            <td style="text-align: center;" data-sortval="6300"><img src='//lkimg.zamimg.com/images/ip_logo.png' width='18' class='champion-price-icon'>6300</td>
            <td style="text-align: center;" data-sortval="10.98">10.98%</td>
            <td style="text-align: center;" data-sortval="48.44">48.44%</td>
            <td style="text-align: center;" data-sortval="18.85">18.85%</td>
            <td style="text-align: center;" data-sorttype="string" data-sortval="Middle Lane">Middle Lane</td>
            <td style="text-align: center;" data-sortval="1323849600">12/14/2011</td>
        </tr> 

I'm having problems extracting the statistics, which are nested in td tags outside of the data-sortval. I imagine that I want to pull ALL the tr tags, but I don't know how to pull the tr tag based off of the one that contains the td tag with data-sortval="Ahri". At that point, I would want to step through the tr tag x times until I reach the first statistic I want, 10.98

At the moment, I'm trying to do a find for the td with data-sortval Ahri, but it doesn't return the rest of the tr.

It might be important to not that all of this is nested inside if a larger tag:

  <table class="clientsort champion-list" width="100%" cellspacing="0" cellpadding="0">
    <thead>
    <tr><th>Champion</th><th>RP Cost</th><th>IP Cost</th><th>Popularity</th><th>Win Rate</th><th>Ban Rate</th><th>Meta</th><th>Released</th></tr>     
    </thead>
    <tbody>

I apologize for the lack of clarity, I'm new with this scraping terminology, but I hope that makes enough sense. Right now, I'm also doing:

main = soup.find('table', {'class':'clientsort champion-list'})

To get only that table

edit:

I typed this for the variable:

for champ in champs:
    a = str(champ)
    print type(a) is str
    td_name = soup.find('td',{"data-sortval":a})

It confirms that a is a string. But it throws this error:

  File "lolrec.py", line 82, in StatScrape
    tr = td_name.parent
AttributeError: 'NoneType' object has no attribute 'parent'

Answer 1

GO LOL!

For commercial purpose, please read the terms of services before scraping.

(1) To scrape a list of heroes, you can do this, which follows a similar logic as you described.

from bs4 import BeautifulSoup
import urllib2
html = urllib2.urlopen('http://www.lolking.net/champions/')
soup = BeautifulSoup(html)
# locate the cell that contains hero name: Ahri 
hero_list = ["Blitzcrank", "Ahri", "Akali"]
for hero in hero_list:
    td_name = soup.find('td', {"data-sortval":hero})
    tr = td_name.parent
    popularity = tr.find_all('td', recursive=False)[3].text
    print hero, popularity

Output

Blitzcrank 12.58%
Ahri 10.98%
Akali 7.52%

Output

10.98%

(2) To scrape all the heroes.

from bs4 import BeautifulSoup
import urllib2
html = urllib2.urlopen('http://www.lolking.net/champions/')
soup = BeautifulSoup(html)
# find the table first
table = soup.find('table', {"class":"clientsort champion-list"})
# find the all the rows
for row in table.find('tbody').find_all("tr", recursive=False):
    cols = row.find_all("td")
    hero = cols[0].text.strip()
    popularity = cols[3].text
    print hero, popularity

Output:

Aatrox 6.86%
Ahri 10.98%
Akali 7.52%
Alistar 4.9%
Amumu 8.75%
...