How I can navigate page a page clicking next button?

Question

I want to navigate into a web with a lof of pages, and I try to click next button to pass to the next page. The web is: https://www.truity.com/search-careers My code is:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.select import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

path = 'C:/Users/.../chromedriver'
driver = webdriver.Chrome(path)
driver.get("https://www.truity.com/search-careers")

while True:
    elements = driver.find_elements_by_xpath('//*[@id="block-system-main"]/div/div[3]/div/table/tbody//a')

    links = []
    for i in range(len(elements)):
        links.append(elements[i].get_attribute('href'))

    for link in links:
        print('navigating to: ' + link)
        driver.get(link)
        # Title
        title.append(driver.title)
        #....

        driver.back()
        
    try:
        driver.find_element_by_xpath('//*[@id="block-system-main"]/div/div[4]/ul/li[11]/a').click()
    except NoSuchElementException:
        break

But my code is not correct. Can you help me? Thanks!

Answer 1

This works for me:

driver.find_element_by_xpath("//*[@id='block-system-main']/div/div[4]/ul/li[11]/a").click()

Alternative:

for i in range(1000):
    try:
        driver.find_element_by_xpath(f"//a[@title='Go to page {i+1}']").click()
    except:
        print('No more pages')
        break

Answer 2

Use driver.find_element_by_link_text()

page_num = 1

while True:

    #insert the code to scrape this page
    #.....
    #.....

    print(f'On page {page_num}')
    
    
    #moving to next page
    page_num+=1
    try:
        driver.find_element_by_link_text(str(page_num)).click()
    except NoSuchElementException:
        print('End of pages')
        break
    time.sleep(3)

Output

Answer 3

To simply loop through all the links and the 17 pages. Wait for all a href elements to be there and grab their href value. Add a time.sleep() incase of stale element error and also wait for the next tag to be clickable. Driver.back() would be an extra step since you just need all the href values and driver.get() to them.

wait = WebDriverWait(driver, 5)
driver.get("https://www.truity.com/search-careers")
title=[]
links=[]
while True:
    elements = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='block-system-main']/div/div[3]/div/table/tbody//a")))
    for elem in elements:
        links.append(elem.get_attribute('href'))
    try:
        wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"li.next > a"))).click()
    except (NoSuchElementException,TimeoutException) as e:
        break
    time.sleep(1)
    
for link in links:
    print('navigating to: ' + link)
    driver.get(link)
    # Title
    title.append(driver.title)
    #....

Imports

from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
from time import sleep