抓取具有表格但表格上的下一步按钮不会更改 url 的网站

Question

I wanted to scrape this link and get the whole table of players:- https://www.nba.com/stats/leaders/?StatCategory=FG3M&PerMode=Totals&Season=2015-16&SeasonType=Regular%20Season

Here, if you click on the next button in the table, the contents of the table changes but the url on the top doesn't change. But the button doesn't have a button tag. It looks like this:-

<a class="stats-table-pagination__next" href="" alt="Next Page" ng-click="nav(1)">       
    <i class="fa fa-angle-right" aria-hidden="true"></i>
</a>

I tried using beautiful soup and selenium to scrape this website but I can't figure out how to navigate to other pages of the table so that I can scrape them too. Please suggest a solution.

Answer 1

1. You can use use google chrome in developer mode and find that json file containing all the data from image that you can see

2. Then go to Network tab and refresh link and go to xhr tab you will find lots of link from that one link contains players information

3. after getting that exact data click on that link copy address and use requests module get json data and extract the information

import requests res=requests.get("https://stats.nba.com/stats/leagueLeaders?LeagueID=00&PerMode=Totals&Scope=S&Season=2015-16&SeasonType=Regular+Season&StatCategory=FG3M") data=res.json() for i in range(len(data['resultSet']['rowSet'])): print(data['resultSet']['rowSet'][i][2])

Output:

Stephen Curry
Klay Thompson
James Harden
Damian Lillard
..

Image: