美湯 findall 沒有返回所有結果
[英]Beautiful soup findall not returning all the results
問題描述
我想在BeautifulSoup中使用find_all通過板球比賽的球數據來刮球。 代碼是：
import requests
from bs4 import BeautifulSoup
url = 'http://www.espncricinfo.com/series/10904/commentary/1075502/south-africa-vs-bangladesh-1st-test-bangladesh-tour-of-sa-2017-18'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
comment = soup.find_all('div', class_ = "over-circle")
print(len(comment))
print(comment[22])
我閱讀了有關此問題的先前問題的答案，幾乎所有內容都涉及使用不同的 html 解析器。 我已經嘗試過lxml, html.parser, html5lib但上面提到的這些（在以前的問題中主要推薦）似乎都沒有給出任何不同的結果。 沒有。 球數顯示為 23，而它應該更多。 輸出：
23
<div class="over-circle low-score" data-reactid="463"><span class="over-score" data-reactid="464">0</span></div>
2 個解決方案
解決方案1
0 2019-06-14 07:34:26
你猜對了。 並非所有數據都一次性加載（因此，您只能看到最初加載的數據）。 您可以實現將循環直到程序到達最后一頁的附加邏輯。
這是其中一個數據頁面的 URL： https://site.web.api.espn.com/apis/site/v2/sports/cricket/10904/playbyplay?contentorigin=espn&event=1075502&page=6&period=4§ion=cricinfo你將需要增加page參數，直到您不斷獲得有效數據。
如果您檢查此 URL 的響應，您將看到這是一個包含 24 個項目的 JSON 文件。
解決方案2
0 已采納 2019-06-14 07:52:36
該頁面是動態的，因此並非全部呈現。 您可以直接訪問源並提取 json 響應，其中還包括總頁數。 獲得總頁數后，您可以使用查詢參數迭代這些頁數，將每個前一頁的頁數相加以獲得所有數據的最終輸出。
我不知道你對什么數據感興趣，但它就在那里。 我將其轉換為數據框，但您可以使用 json 結構做您想做的事情：
然而它是嵌套的。 athletesInvolved列由列出的字典組成。 如果需要，您仍然可以將其標准化/展平（讓我知道您是否也希望這樣做，這很容易做到），但顯然會增加行/列的數量。
import requests
from pandas.io.json import json_normalize
url = 'https://site.web.api.espn.com/apis/site/v2/sports/cricket/10904/playbyplay'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}
payload = {
'contentorigin': 'espn',
'event': '1075502',
'page': '1',
'period': '4',
'section': 'cricinfo'}
# Get inital page of data, including total number of pages to iterate through
response = requests.get(url, headers=headers, params=payload).json()
pageCount = response['commentary']['pageCount']
print ('Total pages: %s\nProcessed page: 1' %(pageCount))
# Store the initial page to jsonData, iterate through the next `pageCount` pages and add that to the list for a final result
jsonData = response
for page in range(2, pageCount+1):
payload = {
'contentorigin': 'espn',
'event': '1075502',
'page': page,
'period': '4',
'section': 'cricinfo'}
response = requests.get(url, headers=headers, params=payload).json()
jsonData['commentary']['items'] = jsonData['commentary']['items'] + response['commentary']['items']
print ('Processed page: %s' %page)
df = json_normalize(jsonData['commentary']['items'])
輸出：198 行中前 5 行的樣本
print (df.head(5).to_string())
athletesInvolved awayScore batsman.athlete.displayName batsman.athlete.fullName batsman.athlete.id batsman.athlete.name batsman.athlete.shortName batsman.faced batsman.fours batsman.runs batsman.sixes batsman.team.abbreviation batsman.team.displayName batsman.team.id batsman.team.name batsman.totalRuns bowler.athlete.displayName bowler.athlete.fullName bowler.athlete.id bowler.athlete.name bowler.athlete.shortName bowler.balls bowler.conceded bowler.maidens bowler.overs bowler.team.abbreviation bowler.team.displayName bowler.team.id bowler.team.name bowler.wickets clock date dismissal.batsman.athlete.displayName dismissal.batsman.athlete.fullName dismissal.batsman.athlete.id dismissal.batsman.athlete.name dismissal.batsman.athlete.shortName dismissal.bowled dismissal.bowler.athlete.displayName dismissal.bowler.athlete.fullName dismissal.bowler.athlete.id dismissal.bowler.athlete.name dismissal.bowler.athlete.shortName dismissal.dismissal dismissal.minutes dismissal.retiredText dismissal.text dismissal.type homeScore id innings.ballLimit innings.balls innings.byes innings.day innings.fallOfWickets innings.id innings.legByes innings.noBalls innings.number innings.remainingBalls innings.remainingOvers innings.remainingRuns innings.runRate innings.runs innings.session innings.target innings.totalRuns innings.wickets innings.wides mediaId otherBatsman.athlete.displayName otherBatsman.athlete.fullName otherBatsman.athlete.id otherBatsman.athlete.name otherBatsman.athlete.shortName otherBatsman.faced otherBatsman.fours otherBatsman.runs otherBatsman.sixes otherBatsman.team.abbreviation otherBatsman.team.displayName otherBatsman.team.id otherBatsman.team.name otherBatsman.totalRuns otherBowler.athlete.displayName otherBowler.athlete.fullName otherBowler.athlete.id otherBowler.athlete.name otherBowler.athlete.shortName otherBowler.balls otherBowler.conceded otherBowler.maidens otherBowler.overs otherBowler.team.abbreviation otherBowler.team.displayName otherBowler.team.id otherBowler.team.name otherBowler.wickets over.actual over.ball over.balls over.byes over.complete over.legByes over.limit over.maiden over.noBall over.number over.overs over.runs over.unique over.wickets over.wide period periodText playType.description playType.id postText preText scoreValue sequence shortText speedKPH speedMPH team.abbreviation team.displayName team.id team.name text
0 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 1 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 1 0 0 0.1 SA South Africa 3 South Africa 0 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 410 0 1 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 0 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.1 1 6 0 False 0 0.0 1 0 1 0.1 0 0.01 2 0 4 4th innings no run 2 <b>2.25pm</b> South Africa gather into a huddl... 0 400001 Morkel to Tamim Iqbal, no run 138.452 86.030 BDESH Bangladesh 25 Bangladesh fullish length ball, angled in from wide of th...
1 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 2 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 2 0 0 0.2 SA South Africa 3 South Africa 0 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 420 0 2 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 0 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.2 2 6 0 False 0 0.0 1 0 1 0.2 0 0.02 2 0 4 4th innings no run 2 0 400002 Morkel to Tamim Iqbal, no run 135.891 84.439 BDESH Bangladesh 25 Bangladesh length ball outside off, Tamim stands tall and...
2 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 3 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 3 0 0 0.3 SA South Africa 3 South Africa 0 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 430 0 3 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 0 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.3 3 6 0 False 0 0.0 1 0 1 0.3 0 0.03 2 0 4 4th innings no run 2 Zahi: "The six went for four? Last ball needs ... 0 400003 Morkel to Tamim Iqbal, no run 140.489 87.296 BDESH Bangladesh 25 Bangladesh fullish, comes into Tamim who flicks it to mid...
3 [{'id': '56194', 'name': 'Tamim Iqbal', 'short... 0 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim 4 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 4 0 0 0.4 SA South Africa 3 South Africa 1 00:00 2017-09-28T10:00 Tamim Iqbal Tamim Iqbal Khan 56194 Tamim Iqbal Tamim True Morne Morkel Morne Morkel 46538 Morne Morkel Morkel True 2 Tamim Iqbal b Morkel 0 (2m 4b 0x4 0x6) SR: 0.00 bowled 0 440 0 4 0 4 1 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 1 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.4 4 6 0 False 0 0.0 1 0 1 0.4 0 0.04 2 0 4 4th innings out 9 0 400004 Morkel to Tamim Iqbal, OUT 136.028 84.524 BDESH Bangladesh 25 Bangladesh bowled him! Morkel strikes first over the chas...
4 [{'id': '373696', 'name': 'Mominul Haque', 'sh... 0 Mominul Haque Mominul Haque 373696 Mominul Haque Mominul 1 0 0 0 BDESH Bangladesh 25 Bangladesh 0 Morne Morkel Morne Morkel 46538 Morne Morkel Morkel 5 0 0 0.5 SA South Africa 3 South Africa 1 00:00 2017-09-28T10:00 Mominul Haque Mominul Haque 373696 Mominul Haque Mominul False Morne Morkel Morne Morkel 46538 Morne Morkel Morkel False 0 NaN 0 450 0 5 0 4 0 199062 0 0 4 0 0.0 424 0.0 0 2 424 0 1 0 0 Imrul Kayes Imrul Kayes 280734 Imrul Kayes Imrul 0 0 0 0 BDESH Bangladesh 25 Bangladesh 0 NaN NaN NaN NaN NaN 0 0 0 NaN NaN NaN NaN NaN 0 0.5 5 6 0 False 0 0.0 1 0 1 0.5 0 0.05 2 0 4 4th innings no run 2 0 400005 Morkel to Mominul Haque, no run 139.982 86.981 BDESH Bangladesh 25 Bangladesh <b>huge appeal for a leg before</b>. Not out s...
美麗的湯-`findAll`不能捕獲SVG中的所有標簽（`ElementTree`可以）
[英]Beautiful Soup - `findAll` not capturing all tags in SVG (`ElementTree` does)
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.