简体   繁体   中英

Unable to scrape some table using beautiful soup

I am using BeautifulSoup to scrape the data from the website link . There are total 9 table with same class name but I am only able to get link 5 tables. What changes should I make in the code so I can scrape all the tables present from the above link?

Below is the code I have used:

def ScrapeSecScreen():
    options = webdriver.ChromeOptions()
    options.add_argument('--ignore-certificate-errors')
    options.add_argument('--incognito')
    options.add_argument('--headless')
    driver = webdriver.Chrome("C:/Users/pralo/Downloads/DE/chromedriver", chrome_options=options)
    driver.get('https://www.panamacompra.gob.pa/Inicio/v2/#!/vistaPreviaCP?NumLc=2022-0-30-0-08-CL-024792&esap=0&nnc=0&it=1')
    sleep(10)
    sourcecode = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML")
    # print(sourcecode)
    soup = BeautifulSoup(sourcecode,"html.parser")
    print(soup)
    l1 = []
    tablelist1=soup.findAll('table',{'class':'table table-condensed table-bordered last-line-table'})
    for tr in tablelist1:
        td = tr.find_all('tr')
        row = [tr.text for tr in td]
        l1.append(row)
    print(l1)
ScrapeSecScreen()

Upon inspection, it turns out that only first five tables match the classes you mentioned. To get all tables, you can omit the class argument or specify a class that all the tables share.

Check this line of code:

soup.find_all('table', class_='table')

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM