Web 刮表数据使用美汤

Question

I am trying to learn web scraping in Python for a project using Beautiful Soup by doing the following: Scraping Kansas City Chiefs active team player name with the college attended. This is the url used https://www.chiefs.com/team/players-roster/ .

After compiling, I get an error saying "IndexError: list index out of range".

I don't know if my set classes are wrong. Help would be appreciated.

import requests
from bs4 import BeautifulSoup

url = "https://www.chiefs.com/team/players-roster/"

r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')

roster_table = soup.find('table', class_ = 'd3-o-table d3-o-table--row-striping d3-o-table--detailed d3-o-table--sortable {sortlist: [[0,0]]}')

for person in roster_table.find_all('tbody'):
    rows = person.find_all('tr')
    for row in rows:
        player_name = row.find('td', class_='sorter-lastname selected"')
        player_university = row.find_all('td', class_='sorter-lastname selected')[7].text
        print(player_name,player_university)

Answer 1

I couldn't find a td with class sorter-lastname selected in the source code. You basically need the last td in each row, so this would do:

for person in roster_table.find_all('tbody'):
    rows = person.find_all('tr')
    for row in rows:
        player_name = row.find('td', class_='sorter-lastname selected"')
        player_university = row.find_all('td')[-1].text

PS. scraping tables is extremely easy in pandas :

import pandas as pd

df = pd.read_html('https://www.chiefs.com/team/players-roster/')
df[0].to_cvs('output.csv')

It may take a bit longer, but the output is impressive, for example the print(df[0]) :

               Player     # Pos    HT   WT  Age Exp            College
0       Josh Pederson   NaN  TE   6-5  235   24   R   Louisiana-Monroe
1   Brandin Dandridge   NaN  WR  5-10  180   25   R   Missouri Western
2       Justin Watson   NaN  WR   6-3  215   25   4       Pennsylvania
3    Jonathan Woodard   NaN  DE   6-5  271   28   3   Central Arkansas
4     Andrew Billings   NaN  DT   6-1  311   26   5             Baylor
..                ...   ...  ..   ...  ...  ...  ..                ...
84   James Winchester  41.0  LS   6-3  242   32   7           Oklahoma
85       Travis Kelce  87.0  TE   6-5  256   32   9         Cincinnati
86        Marcus Kemp  85.0  WR   6-4  208   26   4             Hawaii
87        Chris Jones  95.0  DT   6-6  298   27   6  Mississippi State
88    Harrison Butker   7.0   K   6-4  196   26   5       Georgia Tech

[89 rows x 8 columns]

Answer 2

TL;DR: Two issues to solve: (1) indexing, (2) HTML element-queries

Indexing

The Python Index Operator is represented by opening and closing square brackets: [] . The syntax, however, requires you to put a number inside the brackets.

Example: So [7] applies indexing to the preceding iterable (all found td s), to get the element with index 7. In Python indices are 0-based, so they start with 0 for the first element.

In your statement, you take all found cells as <td> HTML-elements of the specific classes as iterable and want to get the 8th element, by indexing with [7] .

row.find_all('td', class_='sorter-lastname selected')[7]

How to avoid index-errors?

Are you sure there are any td elements found in the row ? If some are found, can we guarantee that it are always at least 8.

In this case, the were apparently less than 8 elements.

That's why Python would raise an IndexError , eg in given script line 15:

Traceback (most recent call last):
  File "<stdin>", line 15, in <module>
    player_university = row.find_all('td', class_='sorter-lastname selected')[7].text
IndexError: list index out of range

Better test on length before indexing:

import requests
from bs4 import BeautifulSoup

url = "https://www.chiefs.com/team/players-roster/"

html = requests.get(url).content
soup = BeautifulSoup(html, 'html.parser')

roster_table = soup.find('table', class_ = 'd3-o-table d3-o-table--row-striping d3-o-table--detailed d3-o-table--sortable {sortlist: [[0,0]]}')

for person in roster_table.find_all('tbody'):
    rows = person.find_all('tr')
    for row in rows:
        print(f"person row: {row}")  # debug-print helps to fix element-query
        player_name = row.find('td', class_='sorter-lastname selected"')
        cells = row.find_all('td', class_='sorter-lastname selected')
        player_university = None   # define a default to avoid NameError
        if len(cells) > 7:   # test on minimum length of 8 for index 7
            player_university = cells[7].text
        print(player_name, player_university)

Element-queries

When the index was fixed, the queried names returned empty results as None, None .

We need to debug (thus I added the print inside the loop) and adjust the queries: (1) for the university-name : If you follow RJ's answer and choose the last cell without any class-condition then a negative index like -1 means from backwards, like here: the last. The number of cells should be at least 1 or greater than 0.

(2) for the player-name : It appears to be in the first cell (also with a CSS-class for sorting), nested either in a link-title <a.. title="Player Name"> or in following sibling as inner text of span > a .

CSS selectors

You may use CSS selectors for that an bs4's select or select_one functions. Then you can select the path like td >? >? > a td >? >? > a td >? >? > a and get the title.

Note: the ? placeholders are left as challenging exercise for you.)

️ Tip: most browsers have an inspector (right click on the element, eg the player-name), then choose "inspect element" and an HTML source view opens selecting the element. Right-click again to "Copy" the element as "CSS selector".

Further Reading

About indexing, and the magic of negative numbers like [-1] :

• AskPython: Indexing in Python - A Complete Beginners Guide

.. a bit further, about slicing :

• Real Python: Indexing and Slicing

Research on Beautiful Soup here:

• Using BeautifulSoup to extract the title of a link
• Get text with BeautifulSoup CSS Selector