Select element using XPATH with Python?

Question

I am trying to determine the number of pages of data generated by the Indian Central Pollution Controal Board. Here is an example of output . Following https://github.com/RachitKamdar/Python-Scraper , I used selenium/python

maxpage = int(browser.find_elements(By.XPATH,"//*[@id='DataTables_Table_0_paginate']/span/a")[-1].text)

but this produces an empty array. I am really not sure what I am doing wrong. Any help would be greatly appreciated. Thanks

Answer 1

You have to add expected condition to wait until the page loaded the data.
You can wait for visibility of element you are using and after that get it's text, like this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//*[@id='DataTables_Table_0_paginate']/span/a")))
maxpage = int(browser.find_elements(By.XPATH,"//*[@id='DataTables_Table_0_paginate']/span/a")[-1].text)

Answer 2

You might want to try getattribute('textContent')

In your case:

maxpage=browser.find_element_by_xpath("(//*[@id='DataTables_Table_0_paginate']/span/a)[last()]").getattribute('textContent')