如何移动到 Python Selenium 的下一页？

Question

我正在尝试为特定站点构建代理抓取工具，但无法移至下一页。

这是我正在使用的代码。

如果您回答我的问题，请向我解释一下您使用的是什么，如果可以的话，如果有任何关于此类代码的好教程，请给我一些：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
#options.headless = True     #for headless
#options.add_argument('--disable-gpu') #for headless and os win

driver = webdriver.Chrome(options=options)

driver.get("https://hidemyna.me/en/proxy-list/")
time.sleep(10) #bypass cloudflare


tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")

for column in cell:
    column = column.text.split(" ")
    print (column[0]+":"+ column[1]) #ip and port

nxt = driver.find_element_by_class_name('arrow_right')
nxt.click()

Answer 1

要移动到下一页，您可以尝试以下解决方案：

• 代码块：

   from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException, WebDriverException options = Options() options.add_argument("start-maximized") options.add_argument("disable-infobars") options.add_argument("--disable-extensions") driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe') driver.get('https://hidemyna.me/en/proxy-list/') while True: try: driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[@class='arrow__right']/a")))) driver.find_element_by_xpath("//li[@class='arrow__right']/a").click() print("Navigating to Next Page") except (TimeoutException, WebDriverException) as e: print("Last page reached") break driver.quit()

• 控制台输出：

   Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page Navigating to Next Page. . . Navigating to Next Page Last page reached

Answer 2

您实际上并没有点击锚点<a>标签。 要导航到下一页，您需要click <a>链接。

您可以像下面这样使用find_element_by_xpath 。

driver.find_element_by_xpath('//*[@id="content-section"]/section[1]/div/div[4]/ul/li[1]/a').click()

您可以按照另一个@Andersson的建议使用 css 选择器，而不是使用 xpath。

Answer 3

下一个按钮往往因网页而异...您必须检查按钮并使用 xpath 或 beaufifulsoup 对其进行处理

通常有“下一页”和“上一页”...将您的 xpath 定位到“下一页”