How to move to the next page on Python Selenium?

Question

I am trying to build a proxy scraper for a specific site, but I'm failing on move to next page.

This is the code that I'm using.

If you answer my question, please, explain me a bit about what you used and if you can, please, if there are any good tutorials for over this kind of code, provide me some:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
#options.headless = True     #for headless
#options.add_argument('--disable-gpu') #for headless and os win

driver = webdriver.Chrome(options=options)

driver.get("https://hidemyna.me/en/proxy-list/")
time.sleep(10) #bypass cloudflare


tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")

for column in cell:
    column = column.text.split(" ")
    print (column[0]+":"+ column[1]) #ip and port

nxt = driver.find_element_by_class_name('arrow_right')
nxt.click()

Answer 1

To move to the next page you can try the following solution:

• Code Block:

   from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException, WebDriverException options = Options() options.add_argument("start-maximized") options.add_argument("disable-infobars") options.add_argument("--disable-extensions") driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe') driver.get('https://hidemyna.me/en/proxy-list/') while True: try: driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[@class='arrow__right']/a")))) driver.find_element_by_xpath("//li[@class='arrow__right']/a").click() print("Navigating to Next Page") except (TimeoutException, WebDriverException) as e: print("Last page reached") break driver.quit()

• Console Output:

   Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page. . . Navigating to Next Page Last page reached

Answer 2

You are not actually clicking the anchor <a> tag. To navigate to next page you need to click on the <a> link.

You can use find_element_by_xpath like below.

driver.find_element_by_xpath('//*[@id="content-section"]/section[1]/div/div[4]/ul/li[1]/a').click()

Instead of using xpath you may use css selector as suggested by another @Andersson .

Answer 3

the next button tend to vary from webpage to webpage...you will have to inspect the button and address it with xpath or beaufifulsoup

There's usually 'next page' and 'previous page'...address your xpath to 'next'