Python: selenium click next button

Question

I am using Python Selenium with ChromeDriver, all are up to date. I am trying click on "Next" button until the last page.

I have tried myself but after few page clicks the script breaks or stops from clicking further. I have made some edits and lost the partial working code.

Here is the html code.

html at the beginning of the page:

<div class="pagination" total="2098" limit="20" offset="1" view="products">
  <ul>
    <li class="disabled page">First
    </li>
    <li class="disabled page">Prev
    </li>
    <li key="1" class="pageLink digital current page">1
    </li>
    <li key="2" class="pageLink digital page">2
    </li>
    <li key="3" class="pageLink digital page">3
    </li>
    <li key="2" class="pageLink page">Next
    </li>
    <li key="105" class="pageLink page">Last
    </li>
  </ul>
</div>

html at the last page:

<div class="pagination" total="6866" limit="20" offset="344" view="products"><ul><li key="1" class="pageLink page">First</li><li key="343" class="pageLink page">Prev</li><li key="342" class="pageLink digital page">342</li><li key="343" class="pageLink digital page">343</li><li key="344" class="pageLink digital current page">344</li><li class="disabled page">Next</li><li class="disabled page">Last</li></ul></div>

Edit:

Python code I tried.

while True:    
next_page_btn = None
next_page_btn = browser.find_elements_by_xpath("/html[1]/body[1]/div[1]/div[2]/div[2]/div[6]/div[1]/div[1]/div[3]/div[1]/div[1]/ul[1]/li[6]")
if len(next_page_btn) < 1:
    print("No more pages left")
    break
else:
    
    element = WebDriverWait(browser, 10).until( 
    EC.presence_of_element_located((By.xpath, "/html[1]/body[1]/div[1]/div[2]/div[2]/div[6]/div[1]/div[1]/div[3]/div[1]/div[1]/ul[1]/li[6]")) 
    )
    element.click()

EDIT 2: Below code is what I am using so far it is working fine except one issue. Even after the last page it keep continuing loading the last page without end. How do we stop when it reaches at the end.

while True:
time.sleep(5)
#wait for pagination to show 
EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'pagination')]")) 
next_page_btn = browser.find_elements_by_xpath("//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]")
if len(next_page_btn) < 1:
    print("No more pages left")
    break
else:
    WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//li[.='Next']"))).click()

Answer 1

Normally the pagination state (the page number) is stored in the URL as a query string, You can simply use a counter and loop through the URLs without worrying about targeting the element correctly

Answer 2

I would use this xpath to identify the next button:

//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]

while True:   
    #wait for pagination to show 
    EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'pagination')]")) 
    )
next_page_btn = browser.find_elements_by_xpath("//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]")
if len(next_page_btn) < 1:
    print("No more pages left")
    break
else:
    element.click()

Or you could simplify by doing try catch:

while True:
    EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'pagination')]"))
    try:
        driver.find_element_by_xpath("//div[contains(@class, 'pagination')]//li[contains(text(), 'Next')]").click()
    except NoSuchElementException:
        print("Element not found")
        break

Answer 3

try using "find_elements_by_link_text" instead of xpath

Answer 4

Try this code to click Next until no pages left

browser.get(URL)

while True:
    ... <scraping code>...
    try:
        WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//li[.='Next']"))).click()
    except:
        print('No more pages to load')
        break