How to execute “javascript:__doPostBack” in a webpage to download pdf files using selenium?

Question

I have tried all the solutions from this very similar post but unfortunately, while I do not get any helpful error and neither do I get any pdf files in my folder.

To change the configuration so that selenium works headless and downloads to a directory I want, I followed this post and this .

However I don't see anything. Also the behaviors are different when executing interactively vs when running a script. When executing interactively I don't see any error but then nothing happens as well. When running a script I get a not so useful error:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, f"a[href*={css_selector}']"))).click()
  File "C----\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

The website in question is here .

The code that I am trying to make working is -

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.headless = True

uri = "http://affidavitarchive.nic.in/CANDIDATEAFFIDAVIT.aspx?YEARID=March-2017+(+GEN+)&AC_No=1&st_code=S24&constType=AC"

driver = webdriver.Firefox(options=options, executable_path=r'C:\\Users\\xxx\\geckodriver.exe')

profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.download.dir', r'C:\\Users\\xxx\\Downloads')
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')

# Function that reads the table in the webpage and extracts the links for the pdfs
def get_links_from_table(uri):
    html = requests.get(uri)
    soup = BeautifulSoup(html.content, 'lxml')
    table = soup.find_all('table')[-1]
    candidate_affidavit_links = []
    for link in table.find_all('a'):
        candidate_affidavit_links.append(link.get('href'))
    return candidate_affidavit_links

candidate_affidavit_links_list = get_links_from_table(uri)

driver.get(uri)

# iterate over the javascript links and try to download the pdf files
for js_link in candidate_affidavit_links_list:
    css_selector = js_link.split("'")[1]
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, f"a[href*={css_selector}']"))).click()
    driver.execute_script(js_link)

Answer 1

If all this can be done with Selenium I would try this:

driver.get(uri)
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "(//table//a)[last()]")))
time.sleep(1)
candidate_affidavit_links = driver.find_elements_by_xpath("(//table//a)[last()]")
for link in candidate_affidavit_links:
    link.click()
    time.sleep(1)

Open the page, wait until at least first link in the table is visible, add some more wait until all the table is surely loaded, get all the a (links) elements to the list, iterate through that list clicking on those elements and putting a delay after each click to make downloading complete.
Possibly you will need to put longer delay after clicking each link to complete downloading file before next downloading is started.
UPD
To disable pop-ups asking to save file etc try this: Instead of just

profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')

put this:

profile.set_preference('browser.helperApps.neverAsk.saveToDisk", "application/csv,application/excel,application/vnd.ms-excel,application/vnd.msexcel,text/anytext,text/comma-separated-values,text/csv,text/plain,text/x-csv,application/x-csv,text/x-comma-separated-values,text/tab-separated-values,data:text/csv')
profile.set_preference('browser.helperApps.neverAsk.saveToDisk", "application/xml,text/plain,text/xml,image/jpeg,application/octet-stream,data:text/csv')
profile.set_preference('browser.download.manager.showWhenStarting',false)
profile.set_preference('browser.helperApps.neverAsk.openFile","application/csv,application/excel,application/vnd.ms-excel,application/vnd.msexcel,text/anytext,text/comma-separated-values,text/csv,text/plain,text/x-csv,application/x-csv,text/x-comma-separated-values,text/tab-separated-values,data:text/csv')
profile.set_preference('browser.helperApps.neverAsk.openFile","application/xml,text/plain,text/xml,image/jpeg,application/octet-stream,data:text/csv')
profile.set_preference('browser.helperApps.alwaysAsk.force', false)
profile.set_preference('browser.download.useDownloadDir', true)
profile.set_preference('dom.file.createInChild', true)

Not sure you need all this, but I have all this and it works for me

Answer 2

This is much simpler in chrome:

driver = webdriver.Chrome()

driver.execute_cdp_cmd("Page.setDownloadBehavior", {"behavior": "allow", "downloadPath": "/path/to/folder"})

driver.get("http://affidavitarchive.nic.in/CANDIDATEAFFIDAVIT.aspx?YEARID=March-2017+(+GEN+)&AC_No=1&st_code=S24&constType=AC")

for a in driver.find_elements_by_css_selector('a[href*=doPostBack]'):
  a.click()