使用python从下拉菜单中抓取所有动态生成的数据的最佳方法

Question

I am creating webscraper that will scrape dynamically generated player data from this website

https://frozenpool.dobbersports.com/frozenpool_linecombo.php

I want to create a loop that will generate the data for a player from a drop-down menu, scrape the data, and then loop through the entire list of players in the drop down menu.

I am curious if using selenium to interact with the site is the best way to do this. But I have also noticed that the URL for every player follows a specific pattern, so I have considered scraping the initial page to collect all the data I need, then using that data to construct a list of URL's, then looping through that list of URL's and treating them as static pages.

Are there python tools that are built for this specific type of web scraping?

Answer 1

It looks like all the information you need is loaded in the request to https://frozenpool.dobbersports.com/frozenpool_linecombo.php . Because it doesn't make additional requests to get more information Selenium is probably a bit overkill for this project and you can stick with BeautifulSoup .

I think you are correct in your idea of looping through each player using the structure URL. If this is a one off scrape then copy the list of players directly from the html of the page to get the player ids, then loop though the following URL, replacing PLAYER_ID with each players identifier.

http://frozenpool.dobbersports.com/frozenpool_linecombo.php?select=F&forward= PLAYER_ID &games=2019-2020%3AR%3A99&period=ALL&situation=ALL