Beautifulsoup：激活網頁按鈕並繼續在新頁面上抓取

Question

我有一個大學項目，需要在線獲取數據。 我想從這個網站獲取一些數據。 https://www.footballdatabase.eu/en/transfers/-/2020-10-03

10 月 3 日，我設法獲得了前 19 行，但后來有 6 頁，我正在努力激活用於加載下一頁的按鈕。

這是按鈕的html代碼：

<a href="javascript:;" class="inactive" onclick="showtransfers('1','2020-10-03','2','full');">2</a>

到目前為止我的代碼：

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'User-Agent': 
           'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

page = "https://www.footballdatabase.eu/en/transfers/-/2020-10-03"
pageTree = requests.get(page, headers=headers)
pageSoup = BeautifulSoup(pageTree.content, 'html.parser')

Players = pageSoup.find_all("span", {"class": "name"})
Team = pageSoup.find_all("span", {"class": "firstteam"})
Values = pageSoup.find_all("span", {"class": "transferamount"})
Values[0].text

PlayersList = []
TeamList = []
ValuesList = []
j=1

for i in range(0,20):
    PlayersList.append(Players[i].text)
    TeamList.append(Team[i].text)
    ValuesList.append(Values[i].text)
    j=j+1
df = pd.DataFrame({"Players":PlayersList,"Team":TeamList,"Values":ValuesList})

非常感謝！

Answer 1

您可以使用requests模塊來模擬 Ajax 調用。 例如：

import requests
from bs4 import BeautifulSoup


data = {
    'date':  '2020-10-03',
    'pid': 1,
    'page': 1,
    'filter': 'full',
}

url = 'https://www.footballdatabase.eu/ajax_transfers_show.php'

for data['page'] in range(1, 7):  # <--- adjust number of pages here.
    soup = BeautifulSoup(requests.post(url, data=data).content, 'html.parser')

    for line in soup.select('.line'):
        name = line.a.text
        first_team = line.select_one('.firstteam').a.text if line.select_one('.firstteam').a else 'Free'
        second_team = line.select_one('.secondteam').a.text if line.select_one('.secondteam').a else 'Free'
        amount = line.select_one('.transferamount').text

        print('{:<30} {:<20} {:<20} {}'.format(name, first_team, second_team, amount))

印刷：

Bruno Amione                   Belgrano             Hellas Vérone        1.7 M€
Ismael Gutierrez               Betis Deportivo      Atlético B           1 M€
Vitaly Janelt                  Bochum               Brentford            500 k€
Sven Ulreich                   Bayern Munich        Hambourg SV          500 k€
Salim Ali Al Hammadi           Baniyas              Khor Fakkan          Prêt
Giovanni Alessandretti         Ascoli U-20          Recanatese           Prêt
Gabriele Bellodi               AC Milan U-20        Alessandria          Prêt
Louis Britton                  Bristol City B       Torquay United       Prêt
Juan Brunetta                  Godoy Cruz           Parme                Prêt
Bobby Burns                    Barrow               Glentoran            Prêt
Bohdan Butko                   Shakhtar Donetsk     Lech Poznan          Prêt
Nicolò Casale                  Hellas Vérone        Empoli               Prêt
Alessio Da Cruz                Parme                FC Groningue         Prêt
Dalbert Henrique               Inter Milan          Rennes               Prêt

...and so on.