如何使用美丽的汤4和蟒蛇和硒循环页面？

Question

I'm Fairly new to Python and using beautiful soup first time though I have some experience with selenium. I am trying to scrape a website (" http://cbseaff.nic.in/cbse_aff/schdir_Report/userview.aspx " ) For all the affiliation number.

The problem is they are on multiple pages( 20 result on 1, total: 21,000+ result)

so, I wish to scrape these in some kind of loop that can iterate over the next page btn, the problem in URL of the web page does not change and thus there is no pattern.

Okay so for this i have tried, google sheet Import HTML/ Import XML method but due to large scale of problem it just hangs. Next I tried python and started reading about scraping using python (I'm doing this for the first time :) ) Some-one on this platform suggested an method

( Python Requests/BeautifulSoup access to pagination )

I am trying to do the same but with little and no success.

Also, to fetch the result we have to first, query the search bar with the keyword "a" --> then click search. Only then the website show result.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by  import By
import time

options = webdriver.ChromeOptions()
options.add_argument("headless")
driver = webdriver.Chrome(executable_path=r"C:\chromedriver.exe",options=options)

driver.get("http://cbseaff.nic.in/cbse_aff/schdir_Report/userview.aspx")
#click on the radio btn
driver.find_element(By.ID,'optlist_0').click()

time.sleep(2)

# Search the query with letter A And Click Search btn
driver.find_element(By.ID,'keytext').send_Keys("a")
driver.find_element(By.ID,'search').click()

time.sleep(2)

next_button = driver.find_element_by_id("Button1")
data = []
try:
    while (next_button):    
        soup = BeautifulSoup(driver.page_source,'html.parser')
        table = soup.find('table',{'id':'T1'}) #Main Table
        table_body = table.find('tbody') #get inside the body
        rows = table_body.find_all('tr') #look for all tablerow
        for row in rows:            
            cols = row.find_all('td')  # in every Tablerow, look for tabledata
                for row2 in cols:
                    #table -> tbody ->tr ->td -><b> --> exit loop. ( only first tr is our required data, print this)

The final outcome I expect is List of all affiliation number across multiple pages.

Answer 1

A minor addition to the code within your while loop:

next_button = 1 #Initialise the variable for the first instance of while loop

while next_button:
    #First scroll to the bottom of the page
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 
    #Now locate the button & click on it
    next_button = driver.find_element(By.ID,'Button1')
    next_button.click()
    ###
    ###Beautiful Soup Code : Fetch the page source now & do your thing###
    ###
    #Adjust the timing as per your requirement
    time.sleep(2)

Note the fact that scrolling to the bottom of the page is important, otherwise an error will pop up claiming 'Button1' element is hidden under the footer. So with the script(in the beginning of the loop), the browser will move down to the bottom of the page. Here, it can see the 'Button1' element clearly. Now, locate the element, perform the click action & then let your Beautiful Soup take over.