使用“美丽汤”刮擦多页-仅显示最后一页
[英]Scrape multiple pages with Beautiful Soup - only show last page
问题描述
我和我的朋友每年都会做一次国际足联选秀，而我一直在寻找一种更新球员信息的快速方法。
我想要从不同页面的播放器中抓取一些信息。 这些页面是：
https://sofifa.com/players?offset=0
https://sofifa.com/players?offset=51
https://sofifa.com/players?offset=101
这是我的代码：
import requests
from bs4 import BeautifulSoup
import re
pages = []
collection = ['0', '51', '101']
for i in collection:
url = 'https://sofifa.com/players?offset=' + str(i)
pages.append(url)
for item in pages:
page = requests.get(item)
soup = BeautifulSoup(page.text, 'html.parser')
table = soup.find('table', {'class': 'table-hover'})
tbody = table.find('tbody')
info = tbody.find_all('tr')
records = []
for infos in info:
nome = infos.find("a", href=re.compile("/player/")).string
numero = infos.find('img').get('id')
idade = infos.find('div' , {'class': 'col-ae'}).string
ponto = infos.find('div' , {'class': 'col-oa'}).string
potencial = infos.find('div' , {'class': 'col-pt'}).string
records.append((nome, numero, idade, ponto, potencial))
import pandas as pd
df = pd.DataFrame(records, columns=['nome', 'numero', 'idade', 'ponto', 'potencial'])
df.to_excel('jogadores.xls', index=False, encoding='utf-8')
事实是，我仅从页面“ 101”获得结果，而未分析前两个页面。 我在这里到处搜寻，但没有运气。
我感到困惑的是，如果我打印（汤），它将向我显示三页中的HTML！
我该如何解决？
这是HTML代码的一个示例：
<table class="table table-hover persist-area">
<thead>...</thead>
<tbody> <tr>
<td> <figure class="avatar">
<img alt="" data-src="https://cdn.sofifa.org/players/4/19/158023.png" data-srcset="https://cdn.sofifa.org/players/4/19/158023@2x.png 2x, https://cdn.sofifa.org/players/4/19/158023@3x.png 3x" src="https://cdn.sofifa.org/players/4/19/158023.png" width="48" height="48" id="158023" class="player-check loaded" srcset="https://cdn.sofifa.org/players/4/19/158023@2x.png 2x, https://cdn.sofifa.org/players/4/19/158023@3x.png 3x" data-was-processed="true"></figure></td>
<td>
<div class="col-name text-ellipsis rtl"><a href="/players?na=52" rel="nofollow" title="Argentina"><img alt="" src="https://cdn.sofifa.org/flags/52.png" data-src="https://cdn.sofifa.org/flags/52.png" data-srcset="https://cdn.sofifa.org/flags/52@2x.png 2x, https://cdn.sofifa.org/flags/52@3x.png 3x" class="flag loaded" style="width:23px;height:17px" srcset="https://cdn.sofifa.org/flags/52@2x.png 2x, https://cdn.sofifa.org/flags/52@3x.png 3x" data-was-processed="true"></a> <a href="/player/158023" title="Lionel Messi">L. Messi</a> <div class="text-ellipsis rtl"><a rel="nofollow" href="/players?pn=21"><span class="pos pos21">CF</span></a> <a rel="nofollow" href="/players?pn=23"><span class="pos pos23">RW</span></a> <a rel="nofollow" href="/players?pn=25"><span class="pos pos25">ST</span></a></div>
</div>
</td> <td class="col text-center" data-col="ae">
<div class="col-digit col-ae">31</div>
</td> <td class="col text-center" data-col="oa">
<div class="col-digit col-oa"><span class="label p94">94</span></div>
</td> <td class="col text-center" data-col="pt">
<div class="col-digit col-pt"><span class="label p94">94</span></div>
</td> <td>
<div class="col-name text-ellipsis rtl"> <figure class="avatar avatar-sm transparent">
<img alt="" class="team loaded" data-src="https://cdn.sofifa.org/teams/2/19/light/241.png" data-srcset="https://cdn.sofifa.org/teams/2/19/light/241@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/241@3x.png 3x" src="https://cdn.sofifa.org/teams/2/19/light/241.png" width="24" height="24" srcset="https://cdn.sofifa.org/teams/2/19/light/241@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/241@3x.png 3x" data-was-processed="true">
</figure>
<a href="/team/241">FC Barcelona</a> <div class="subtitle text-ellipsis rtl">2004 ~ 2021</div>
</div>
</td><th class="gap"></th> <td class="col text-center" data-col="vl">
<div class="col-digit col-vl">€110.5M</div>
</td> <td class="col text-center" data-col="wg">
<div class="col-digit col-wg">€565K</div>
</td><th class="gap"></th> <td class="col text-center" data-col="tt">
<div class="col-digit col-tt">2195</div>
</td> <td class="gap"></td>
<td> <div class="col-comments text-right text-ellipsis rtl">0.8K / 32.9K </div></td></tr> <tr>
<td> <figure class="avatar">
<img alt="" data-src="https://cdn.sofifa.org/players/4/19/20801.png" data-srcset="https://cdn.sofifa.org/players/4/19/20801@2x.png 2x, https://cdn.sofifa.org/players/4/19/20801@3x.png 3x" src="https://cdn.sofifa.org/players/4/19/20801.png" width="48" height="48" id="20801" class="player-check loaded" srcset="https://cdn.sofifa.org/players/4/19/20801@2x.png 2x, https://cdn.sofifa.org/players/4/19/20801@3x.png 3x" data-was-processed="true"></figure></td>
<td>
<div class="col-name text-ellipsis rtl"><a href="/players?na=38" rel="nofollow" title="Portugal"><img alt="" src="https://cdn.sofifa.org/flags/38.png" data-src="https://cdn.sofifa.org/flags/38.png" data-srcset="https://cdn.sofifa.org/flags/38@2x.png 2x, https://cdn.sofifa.org/flags/38@3x.png 3x" class="flag loaded" style="width:23px;height:17px" srcset="https://cdn.sofifa.org/flags/38@2x.png 2x, https://cdn.sofifa.org/flags/38@3x.png 3x" data-was-processed="true"></a> <a href="/player/20801" title="C. Ronaldo dos Santos Aveiro">Cristiano Ronaldo</a> <div class="text-ellipsis rtl"><a rel="nofollow" href="/players?pn=25"><span class="pos pos25">ST</span></a> <a rel="nofollow" href="/players?pn=27"><span class="pos pos27">LW</span></a></div>
</div>
</td> <td class="col text-center" data-col="ae">
<div class="col-digit col-ae">33</div>
</td> <td class="col text-center" data-col="oa">
<div class="col-digit col-oa"><span class="label p94">94</span></div>
</td> <td class="col text-center" data-col="pt">
<div class="col-digit col-pt"><span class="label p94">94</span></div>
</td> <td>
<div class="col-name text-ellipsis rtl"> <figure class="avatar avatar-sm transparent">
<img alt="" class="team loaded" data-src="https://cdn.sofifa.org/teams/2/19/light/45.png" data-srcset="https://cdn.sofifa.org/teams/2/19/light/45@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/45@3x.png 3x" src="https://cdn.sofifa.org/teams/2/19/light/45.png" width="24" height="24" srcset="https://cdn.sofifa.org/teams/2/19/light/45@2x.png 2x, https://cdn.sofifa.org/teams/2/19/light/45@3x.png 3x" data-was-processed="true">
</figure>
<a href="/team/45">Juventus</a> <div class="subtitle text-ellipsis rtl">2018 ~ 2022</div>
</div>
</td><th class="gap"></th> <td class="col text-center" data-col="vl">
<div class="col-digit col-vl">€77M</div>
</td> <td class="col text-center" data-col="wg">
<div class="col-digit col-wg">€405K</div>
</td><th class="gap"></th> <td class="col text-center" data-col="tt">
<div class="col-digit col-tt">2228</div>
</td> <td class="gap"></td>
<td> <div class="col-comments text-right text-ellipsis rtl">0.8K / 40.1K </div></td></tr></tbody>
</table>
1 个解决方案
解决方案1
0 已采纳 2018-10-09 20:05:17
因为G_M已经指出了缩进。 请尝试这种方式：
import requests
from bs4 import BeautifulSoup
import re
pages = []
records = []
collection = ['0', '51', '101']
for i in collection:
url = 'https://sofifa.com/players?offset=' + str(i)
pages.append(url)
for item in pages:
page = requests.get(item)
soup = BeautifulSoup(page.text, 'html.parser')
table = soup.find('table', {'class': 'table-hover'})
tbody = table.find('tbody')
info = tbody.find_all('tr')
for infos in info:
nome = infos.find("a", href=re.compile("/player/")).string
numero = infos.find('img').get('id')
idade = infos.find('div' , {'class': 'col-ae'}).string
ponto = infos.find('div' , {'class': 'col-oa'}).string
potencial = infos.find('div' , {'class': 'col-pt'}).string
records.append((nome, numero, idade, ponto, potencial))
print(nome,numero,idade,ponto,potencial)
import pandas as pd
df = pd.DataFrame(records, columns=['nome', 'numero', 'idade', 'ponto', 'potencial'])
df.to_excel('jogadores.xls', index=False, encoding='utf-8')
使用Beautiful Soup和Python刮擦多个搜索页面的结果
[英]Scrape results of multiple search pages with Beautiful Soup and Python
如何使用 Beautiful Soup BS4 在雅虎财经上抓取多个页面
[英]How to scrape multiple pages on Yahoo Finance with Beautiful Soup BS4
用 bs4 Beautiful Soup 刮多页 - 只刮第一页
[英]Scraping multiple pages with bs4 Beautiful Soup - only scrapes the first page
如何使用python和美丽的汤将html页面拆分为多个页面
[英]How to split a html page to multiple pages using python and beautiful soup
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.