简体   繁体   中英

Can't grab all the pdf links within a table from a webpage

I've written a script in python in combination with selenium to scrape different pdf links generated upon clicking on the different numbers, as in 110015710 , 110015670 etc located within a table from a webpage.

Site link

My script can click on those links, reveal the pdf files but parse only 5 of them out of many.

How can I get them all?

I've tried so far:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "replace_with_above_link"

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)

[driver.execute_script("arguments[0].click();",item) for item in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"tr.Iec")))]
for elem in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".IecAttachments li a[href$='.pdf']"))):
    print(elem.get_attribute("href"))
driver.quit() 

when you click the element it will doing XHR to request for pdf links, add delay after every click.

for item in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"tr.Iec"))):
    driver.execute_script("arguments[0].click();",item)
    time.sleep(1)

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM