如何使用 Python 和 BeautifulSoup 从 cricinfo 中提取玩家姓名

Question

I'm learning beautiful soup. I want to extract the player names ie the playing eleven for both teams from cricinfo.com. The exact link is " https://www.espncricinfo.com/series/13266/scorecard/439146/west-indies-vs-south-africa-1st-t20i-south-africa-tour-of-west-indies-2010 " The problem is that the website only displays the players under class "wrap batsmen" if they have batted. Otherwise they are placed under the class "wrap dnb". I want to extract all the players irrespective of whether they have batted or not. How I can maintain two arrays (one for each team) that will dynamically search for players in "wrap batsmen" and "wrap dnb" (if required)?

This is my attempt:

from urllib.request import urlopen
from bs4 import BeautifulSoup
import pandas as pd
years = []
# Years we will be analyzing
for i in range(2010, 2018):
    years.append(i)

names = []


# URL page we will scraping (see image above)
url = "https://www.espncricinfo.com/series/13266/scorecard/439146/west-indies-vs-south-africa-1st-t20i-south-africa-tour-of-west-indies-2010"
# this is the HTML from the given URL
html = urlopen(url)
soup = BeautifulSoup(html, features="html.parser")


for a in range(0, 1):
    names.append([a.getText() for a in soup.find_all("div", class_="cell batsmen")[1:][a].findAll('a', limit=1)])

soup = soup.find_all("div", class_="wrap dnb")
print(soup[0])

Answer 1

While this is possible with BeautifulSoup, it's not the best tool for the job. All that data (and much more) is available through the API. Simply pull that and then you can parse the json to get what you want (and more). Here's a quick script though to get the 11 players for each team:

You can get the api url by using dev tools (Ctrl-Shft-I) and seeing what requests the browser makes (look at Network -> XHR in the side panel. you may need to click around to view it make the request/call)

import requests

url = 'https://site.web.api.espn.com/apis/site/v2/sports/cricket/13266/summary'

payload = {
'contentorigin': 'espn',
'event': '439146',
'lang': 'en',
'region': 'gb',
'section': 'cricinfo'}

jsonData = requests.get(url, params=payload).json()

roster = jsonData['rosters']

players = {}
for team in roster:
    players[team['team']['displayName']] = []
    for player in team['roster']:
        playerName = player['athlete']['displayName']
        players[team['team']['displayName']].append(playerName)

Output:

print (players)
{'West Indies': ['Chris Gayle', 'Andre Fletcher', 'Dwayne Bravo', 'Ramnaresh Sarwan', 'Narsingh Deonarine', 'Kieron Pollard', 'Darren Sammy', 'Nikita Miller', 'Jerome Taylor', 'Sulieman Benn', 'Kemar Roach'], 'South Africa': ['Graeme Smith', 'Loots Bosman', 'Jacques Kallis', 'AB de Villiers', 'Jean-Paul Duminy', 'Johan Botha', 'Alviro Petersen', 'Ryan McLaren', 'Roelof van der Merwe', 'Dale Steyn', 'Charl Langeveldt']}

See below: