How to use Selenium to change the page from “1” to “All” on nba.stats.com

Question

As of now my web scraper works to scrape all of the NBA player names off of the first page of the table (50 rows). I am quite new to this stuff, and I am unsure how to navigate to the "1" and change the value to "All" for the pages to display. My end goal is to scrape all 488 entries at once, instead of only being able to scrape 50. Would I just have to create another driver.execute_script() as I did with the scroll? Thanks.

Here is my code as of now:

from selenium import webdriver
from bs4 import BeautifulSoup
import time
import os

driver = webdriver.Firefox(executable_path="/Users/myusername/Documents/geckodriverfolder/geckodriver")
driver.get('https://stats.nba.com/players/traditional/?PerMode=Totals&sort=NBA_FANTASY_PTS&dir=-1&Season=2017-18&SeasonType=Regular%20Season')
time.sleep(5)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
html = driver.page_source

soup = BeautifulSoup(html, 'lxml')

for test in soup.find_all('td', class_='player'):
player_name = test.a.text
print(player_name)

Answer 1

Here is what I came up with:

driver.find_element_by_xpath("//select/option[text()='All']").click()

I tried to use Select from Selenium, but there was no elementID for the select tag so I used the xpath route.

I would be interested if someone was able to do this with Select without an elementID (if that is possible).

Hope this helps!

Answer 2

This is how it can work with Select :

select = Select(driver.find_element_by_css_selector('.stats-table- pagination__select'))
select.select_by_visible_text("All")

or by index:

select.select_by_index(0)