如何使用 Selenium 遍历 URL 进行抓取

Question

The following is the code I have so far to scrape from https://n.rivals.com/state_rankings/2021/alabama . I want the code to loop through the address with replacing all the states where alabama is. Ideally, I would also like to be able to change the year for future use. What am I doing wrong with how i defined url and year/state1?

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

TIMEOUT = 5

driver = webdriver.Firefox()
driver.set_page_load_timeout(TIMEOUT)

url = f"https://n.rivals.com/state_rankings/{year}/{state1}"
year = "2021"
state1 = "alabama"

try:
    driver.get(url)
except TimeoutException:
    pass

first_names = driver.find_elements_by_class_name('first-name')
first_names = [name.text for name in first_names]

last_names = driver.find_elements_by_class_name('last-name')
last_names = [name.text for name in last_names]

for first, last in zip(first_names, last_names):
    print(first, last)

player_positions = driver.find_elements_by_class_name('pos')
player_positions = [position.text for position in player_positions]

for position in player_positions:
    print(position)

data = driver.find_elements_by_xpath('//div[@class="break-text ng-binding ng-scope"]')
for d in data:
    location, highschool = d.text.strip().split('\n')
    city, state = location.split(',')
    print(city)
    print(state)
    print(highschool)

commit_status = driver.find_elements_by_class_name('school-name')
commit_status = [commit.text for commit in commit_status]

for commit in commit_status:
    print(commit)

driver.close()

Answer 1

You have to create the variables before you refer to them like so:

year = "2021"
state1 = "alabama"
url = f"https://n.rivals.com/state_rankings/{year}/{state1}"

Then to make it loop for many states, you would do the following:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException

TIMEOUT = 5

driver = webdriver.Firefox()
driver.set_page_load_timeout(TIMEOUT)

def rivals_scrape(state, year):
    url = f"https://n.rivals.com/state_rankings/{year}/{state}"

    try:
        driver.get(url)
    except TimeoutException:
        pass

    first_names = driver.find_elements_by_class_name('first-name')

    ... rest of code ...

    for commit in commit_status:
        print(commit)

states = ["alabama", "georgia","texas"]

for state in states:
    rivals_scrape(state, "2021")

driver.close()