Trouble clicking on the button for the next page

Question

I've written some code in python in combination with selenium. I intended to parse the table from a webpage. I've got it working. However, trouble comes up when i try to click on the next page button. The scraper only parse the table from the first page and instead of clicking the next button it quits without throwing any error. So, i can't understand what I'm missing.

Here is the full code for your consideration:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

driver.get("https://toolkit.financialexpress.net/santanderam")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'table.fe-datatable')))

tab_data = driver.find_element_by_css_selector('table.fe-datatable')

while True:
    wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'tr')))
    list_rows = [[cell.text for cell in row.find_elements_by_css_selector('td')]
                 for row in tab_data.find_elements_by_css_selector('tr')]
    for data in list_rows:
        print(data)

    try:
        driver.find_element_by_css_selector('a.ui-paging-next').click()
    except:
        break

driver.quit()

Elements within which the next-page button exists:

<div class="pagination ui-widget"><span class="ui-paging-current ui-state-default ui-state-disabled ui-corner-all ui-paging-prev">Prev</span><span class="ui-paging-current ui-state-default ui-state-disabled ui-state-highlight ui-corner-all">1</span><a class="ui-paging-button ui-state-default ui-corner-all" href="#">2</a><a class="ui-paging-button ui-state-default ui-corner-all" href="#">3</a><a class="ui-paging-button ui-state-default ui-corner-all" href="#">4</a><span class="ui-state-default ui-corner-all ui-state-disabled ui-paging-ellipse">...</span><a class="ui-paging-button ui-state-default ui-corner-all ep" href="#">7</a><a class="ui-paging-button ui-state-default ui-corner-all ui-paging-next" href="#">Next</a></div>

Answer 1

@Grasshopper has already provided with a solution, but I'll try to give more details for you to understand why your code fails

There are two links with the same HTML code present in page source: the first is hidden, second (the one that you need) is not.

You can check it with

print(len(driver.find_elements_by_css_selector('a.ui-paging-next')))

While css-selector or XPath returns you simply the first occurence, search by link text returns link with the visible text only:

print(len(driver.find_elements_by_link_text('Next')))

That's why your find_element_by_css_selector(...) code doesn't work, but find_element_by_link_text(...) does.

Also note that line

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'table.fe-datatable')))

should already return you required element, so there is no need in

tab_data = driver.find_element_by_css_selector('table.fe-datatable')

Just use

tab_data = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'table.fe-datatable')))

To avoid getting StaleElementReferenceException you should re-define your tab_data on each iterarion as tab_data defined on first page will not be accessible on the next page. Just put tab_data definition inside the while loop

UPDATE

In your code try to replace

try:
    driver.find_element_by_link_text('Next').click()
except:
    break

with

first_row = driver.find_element_by_css_selector('table.fe-datatable tr.odd').text
try:
    driver.find_element_by_link_text('Next').click()
except:
    break
wait.until(lambda driver: driver.find_element_by_css_selector('table.fe-datatable tr.odd').text != first_row)