嘗試使用 python 為該網站上的玩家抓取運動數據

Question

這是我正在使用的網頁網址： https : //www.playerprofiler.com/nfl/christian-mccaffrey/

我要查找的數據存儲在以下 html 標簽中： <div class='c-metric-table'>

我使用以下 python 代碼嘗試抓取數據：

import bs4 
from bs4 import BeautifulSoup as soup
my_test_url = 'https://www.playerprofiler.com/nfl/christian-mccaffrey/' 
import requests
res = requests.get(my_test_url)
page_soup = soup(res.content, 'html.parser')

每當我使用：

len(page_soup.find_all('div', class_='c-metric-table'))

終端在應該返回 5 或 10 時返回 0（我不確定）。

不知道我做錯了什么。 請幫忙！

Answer 1

不需要硒，你可以從 api 中提取它。 很酷的網站，我第一次看到它。

import requests
import pandas as pd

url = 'https://www.playerprofiler.com/wp-admin/admin-ajax.php'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36'}
payload = {
'action': 'playerprofiler_api',
'endpoint': '/player/CM-1225'}

jsonData = requests.get(url, headers=headers, params=payload).json()

data = jsonData['data']['Player']['Performance Metrics']['2019']
teamData = data['Team']

del data['Team']

df = pd.DataFrame([data]).merge(pd.DataFrame([teamData]), how='left', left_index=True, right_index=True)

輸出：

print(df.to_string())
  Average Defenders In The Box Average Defenders In The Box Rank Base Front Carry Rate Base Front Carry Rate Rank Base Front Yards Per Carry Base Front Yards Per Carry Rank Big Hits Big Hits Per Game Big Hits Rank Breakaway Run Rate Breakaway Run Rate Rank Breakaway Runs Breakaway Runs Per Game Breakaway Runs Per Game Rank Breakaway Runs Rank Carries Carries Inside 10 Carries Inside 10 Per Game Carries Inside 5 Carries Per Game Carries Per Game Rank Carries Rank CarriesInside 5 Per Game Catch Rate Catch Rate Rank Catchable Target Rate Catchable Target Rate Rank Catchable Targets Dominator Rating Dominator Rating Rank Drop Rate Drop Rate Rank Drops Drops Per Game Drops Per Game Rank Drops Rank Endzone Target Share Endzone Target Share Rank Endzone Targets Evaded Tackles Evaded Tackles Per Game Evaded Tackles Per Game Rank Evaded Tackles Rank Expected Fantasy Points Expected Fantasy Points Per Game Expected Fantasy Points Per Game Rank Expected Fantasy Points Rank Expected Touchdown Differential Expected Touchdowns Fantasy Points Fantasy Points Per Game Fantasy Points Per Game Differential Fantasy Points Per Game Rank Fantasy Points Per Opportunity Fantasy Points Per Opportunity Rank Fantasy Points Per Snap Fantasy Points Per Snap Rank             FullName Game Script Game Script Rank Games Games Played Rank Goal Line Carries Goal Line Carries Per Game Goal Line Carries Per Game Rank Goal Line Carries Rank Juke Rate Juke Rate Rank Light Front Carry Rate Light Front Carry Rate Rank Light Front Yards Per Carry Light Front Yards Per Carry Rank Normalized Fantasy Points Per Game Normalized Fantasy Points Per Game Rank Opportunities Opportunity Share Opportunity Share Rank Pace of Play Pace of Play Rank Pass Routes Pass Routes Per Game Pass Routes Per Game Rank Pass Snaps Production Premium Production Premium Rank QB Rating When Targeted QB Rating When Targeted Rank Receiving TDs Receiving Yards Receiving Yards Per Game Receiving Yards Per Game Rank Receiving Yards Rank Receptions Receptions Per Game Receptions Per Game Rank Receptions Rank Red Zone Carries Red Zone Carries Per Game Red Zone Carries Per Game Rank Red Zone Carries Rank Red Zone Catch Rate Red Zone Catch Rate Rank Red Zone Opportunities Red Zone Opportunity Share Red Zone Opportunity Share Rank Red Zone Receptions Red Zone Receptions Rank Red Zone Snap Share Red Zone Snap Share Rank Red Zone Snaps Red Zone TD Conversion Rate Red Zone TD Conversion Rate Rank Red Zone Target Share Red Zone Target Share Rank Red Zone Targets Red Zone Targets Rank Red Zone Touches Red Zone Touches Per Game Red Zone Touches Per Game Rank Red Zone Touches Rank Route Participation Route Participation Rank Run Blocking Efficiency Run Blocking Efficiency Rank Run Snaps Rush Yards Per Game Rush Yards Per Game Rank Rushing Touchdowns Rushing Touchdowns Rank Rushing Yards Rushing Yards Rank Season Shotgun Carry Rate Shotgun Carry Rate Rank Shotgun Yards Per Carry Shotgun Yards Per Carry Rank Slot Rate Slot Rate Rank Slot Snaps Slot YPR Slot YPR Rank Slot YPT Slot YPT Rank Snap Share Snap Share Rank Snap-weighted Game Script Snap-weighted Game Script Rank Snaps Stacked Front Carry Rate Stacked Front Carry Rate Rank Stacked Front Yards Per Carry Stacked Front Yards Per Carry Rank Stuffed Run Rate Stuffed Run Rate Rank Stuffed Runs Stuffed Runs Rank Target Share Target Share Rank Targets Targets Inside 10 Targets Inside 10 Per Game Targets Inside 5 Targets Inside 5 Per Game Targets Per Game Targets Per Game Rank Targets Rank Team Run Plays Team Run Plays Rank Total Touchdowns Total Touchdowns Rank Total Touches Total Yards Total Yards Per Game Total Yards Per Game Rank Total Yards Rank Touchdown Rate Touchdown Rate Rank True Yards Per Carry True Yards Per Carry Rank Under Center Carry Rate Under Center Carry Rate Rank Under Center Yards Per Carry Under Center Yards Per Carry Rank    VOS VOS Rank Weekly Volatility Weekly Volatility Rank Weighted Opportunities Weighted Opportunities Per Game Weighted Opportunities Per Game Rank Weighted Opportunities Rank Yards After Catch Yards After Catch Per Game Yards After Catch Per Reception Yards After Catch Per Target Yards After Catch Rank Yards Created Yards Created Per Game Yards Created Per Game Rank Yards Created Per Touch Yards Created Per Touch Rank Yards Created Rank Yards Per Carry Yards Per Carry Rank Yards Per Opportunity Yards Per Reception Yards Per Route Run Yards Per Route Run Rank Yards Per Target Yards Per Target Rank Yards Per Touch Yards Per Touch Rank               Name                                          Permalink            Team_ID
0                          6.4                               #54                 32.1%                        #41                        5.1                              #9        0               0.0           #61               4.2%                     #23             12                     0.8                           #7                  #6     287                32                          2               20             17.9                    #6           #4                      1.2      81.1%             #13                 86.7%                        #20               124            45.8%                    #2      2.1%            #35     3           0.19                 #20        #20                 4.2%                       #10               1             87                     5.4                           #8                  #3                   494.4                             30.9                                    #1                           #1                           +0.89                18.1          469.2                    29.3                                 -1.6                           #1                           1.09                                 #26                    0.47                          #10  Christian McCaffrey       -7.27              #31    16                 1                14                        0.9                              #2                     #1     21.6%            #34                  53.0%                         #15                         4.8                              #24                               27.7                                      #1           430             91.5%                     #1         2.42                #1         405                 25.3                        #1        647              +23.3                     #12                   102.4                          #12             4            1005                     62.8                            #1                   #1        116                 7.2                       #7              #1               56                       3.5                             #3                    #2               71.4%                      #19                     70                      95.9%                              #3                  10                       #2               86.8%                       #6            131                       20.0%                              #20                 21.9%                         #2               14                    #2               66                       4.1                             #2                    #2               64.0%                       #2                    89.9                           #5       356                86.7                       #4                 15                      #3          1387                 #3   2019              47.0%                     #19                     3.7                          #50      3.6%            #11         36      8.1            #7      5.9            #7      98.4%              #1                     -7.27                            #31  1003                    13.6%                           #40                           4.6                                #11            17.4%                   #45           50                #7        23.7%                #1     143                 6                        0.4                3                       0.2              8.9                    #1           #1           24.1                 #24               19                    #1           403        2392                149.5                        #1               #1           4.7%                 #16                  4.4                       #16                   53.3%                          #38                          5.8                                #4  +12.6       #1              10.0                    #10                  396.7                            24.8                                   #1                          #1               979                       61.2                             8.4                          6.8                     #1           415                   25.9                         #14                    1.03                          #52                #12             4.8                  #12                   5.6                 8.7                2.48                       #9              7.0                    #9             5.9                   #6  Carolina Panthers  https://www.playerprofiler.com/depth-charts/ca...  carolina-panthers

編輯

要獲得其他玩家，只需更改有效負載中的玩家 ID。 您從另一個 api 獲取玩家 ID，然后您想過濾它，可以遍歷這些 ID，或者只是手動拉出您想要的 1：

import requests
import pandas as pd

# Get player IDs
url = 'https://www.playerprofiler.com/api/v1/players'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36'}
jsonData = requests.get(url, headers=headers).json()
players = jsonData['data']['Players']

rows = []
for player in players:
    row = {}
    print ('Aquired: %s' %player['Full Name'])
    url = 'https://www.playerprofiler.com/wp-admin/admin-ajax.php'
    payload = {
    'action': 'playerprofiler_api',
    'endpoint': '/player/%s' %player['Player_ID']}
    season = '2019'
    
    jsonData = requests.get(url, headers=headers, params=payload).json()
    if 'Performance Metrics' in jsonData['data']['Player'].keys():
        if season in jsonData['data']['Player']['Performance Metrics'].keys():
            data = jsonData['data']['Player']['Performance Metrics'][season]
        else:
            continue
    else:
        continue
    
    for each in [data, player]:
        row.update(each)
    rows.append(row)
       
df = pd.DataFrame(rows)

rb_df = df[df['Position'] == 'RB']