无法刮表 BeautifulSoup

Question

I'm trying to scrape the following table from this URL: https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show

This is my code:

import requests
from bs4 import BeautifulSoup

url = "https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show"
r = requests.get(url)

soup = BeautifulSoup(r.content, "lxml")

table = soup.find("table")
for row in table.findAll("tr"):
    print([i.text for i in row.findAll("td")])

However, my variable table returns None, even though there is clearly a table tag in the HTML code of the website. How do I get it?

Answer 1

The webpage is loaded dynamically and relies on JavaScript, therefore requests won't support it. You could use another parser library such as selenium .

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()

url = "https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show"

driver.get(url)

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.TAG_NAME, 'table')))

table = driver.find_element(By.TAG_NAME, 'table')

table_html = table.get_attribute('innerHTML')

# print('table html:', table_html)

for tr_web_element in table.find_elements(By.TAG_NAME, 'tr'):
    for td_web_element in tr_web_element.find_elements(By.TAG_NAME, 'td'):
        print(td_web_element.text)

driver.close()

Or see this answer to incorporate Selenium with BeautifulSoup.