pandas.read_html 和 BeautifulSoup 都无法在网页上找到所有表格

Question

I am trying to get the 3rd and 6th tables from a webpage ( https://www.pro-football-reference.com/years/2021/ ) but pandas.read_html and BeautifulSoup are both only finding the first two tables on the page . 这是我尝试过的。

 url = 'https://www.pro-football-reference.com/years/2021/' data_pd = pd.read_html(url) print(len(data_pd))

Output：

 2

并且

url = 'https://www.pro-football-reference.com/years/2021/' soup = BeautifulSoup(requests.get(url).content, 'html.parser') for table in soup.find_all('table'): print(table.get('class'))

Output：

 ['sortable', 'stats_table'] ['sortable', 'stats_table']

我猜这与网页的格式化方式有关，但是我能做些什么来获取我需要的表格吗？

Answer 1

是的，您可以使用 Selenium 让页面呈现然后拉入 html。 但是，如果可以避免开销，我会尽量避免 Selenium。

更好的选择是通过简单的请求，static html 确实有其他表格，但在评论中。 你可以这样做 a) BeautifulSoup 确实有能力提取评论然后解析这些表。 或者干脆删除评论标签，然后解析。

 import requests import pandas as pd url = 'https://www.pro-football-reference.com/years/2021/' response = requests.get(url).text.replace("<,--".""),replace("-->"."") data_pd = pd read_html(response) print(len(data_pd))

Output：

 print(len(data_pd)) 13

或使用 BEautifulSoup 共同通过评论：

 import requests from bs4 import BeautifulSoup, Comment import pandas as pd url = 'https://www.pro-football-reference.com/years/2021/' result = requests.get(url).text data = BeautifulSoup(result, 'html.parser') comments = data.find_all(string=lambda text: isinstance(text, Comment)) data_pd = pd.read_html(url) for each in comments: if '<table' in str(each): data_pd.append(pd.read_html(str(each))[0]) print(len(data_pd))