在 Python 中，當元素列表的長度發生變化時，如何循環遍歷給定 Xpath 組的所有可點擊元素

Question

我正在嘗試生成代碼，該代碼將在紐約公共雇員工資的這個網站上循環播放，並單獨 select 每個可用的"Subagency/Type"選項。 挑戰在於選項的數量每年都在變化。 我已經想到了幾種方法可以做到這一點，但一直無法弄清楚其中的任何一種。 這些元素的 XPATH 遵循以下模式

所以在我意識到選項的數量每年都在變化之前，我一直在使用這樣的循環：

for box in tqdm(range(1,74)):
    link_to_use = '//*[@id="subagencyGroup"]/ul/li[' + str(box) + ']/span[1]'
    WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, link_to_use)))
    driver.find_element(By.XPATH, link_to_use).click()

然后我將對結果執行一些代碼並通過取消單擊該元素來完成迭代，以便下一次迭代只會單擊下一個元素。 我的想法是也許我可以使用 beautifulsoup 來獲取所有 li[N] 值並在頁面上找到它們的最大值 N，然后使用 that+1 作為我的范圍迭代器的上限值。

但后來我開始尋找更直接的途徑，即從頁面中獲取要單擊的元素列表，因此代碼看起來更像：

elem = driver.find_elements_by_xpath('//*[@id="subagencyGroup"]')
all_li = elem.find_elements_by_tag_name("li")

for el in tqdm(all_li):
    WebDriverWait(driver, 10).until(EC.presence_of_element_located(el)
    el.click()

但這似乎因某種我不知道的原因而停滯 - 盡管成功單擊了第一個選項。

完整的代碼如下，我將不勝感激任何指導。

# check if "page next" button is disabled
def check_for_break_condition(driver):

    page_next_button = driver.find_element(By.ID, "data_loader")
    styles = page_next_button.get_attribute("style")
    if('none' in styles):
        return True
    else:
        return False

# Loop Through Years 
for yr in range(1, 15):

    # Store URL to Use
    url = 'https://www.seethroughny.net/payrolls'

    # Initialize WebDriver
    options = Options()
    # options.headless = True
    options.add_argument("start-maximized")
    options.add_argument("disable-infobars")
    options.add_argument("--disable-extensions")
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-application-cache')
    options.add_argument('--disable-gpu')
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--window-size=1560,840") 
    driver = webdriver.Chrome('C:/Windows/System32/chromedriver.exe', options=options)
    driver.get(url)

    try:
        driver.find_element(By.XPATH, '//*[@id="donate-banner"]/button/span').click()
    except:
        pass

    WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[@id="stny_payYear_2020"]/div')))
    driver.find_element(By.CSS_SELECTOR, "#yearGroupHeading a").click()

    WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[@id="stny_payYear_2020"]/div')))
    driver.find_element(By.XPATH, '//*[@id="stny_payYear_2020"]/div').click()

    yr_path = '//*[@id="yearGroup"]/ul/li[' + str(yr) + ']'
    WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, yr_path)))
    driver.find_element(By.XPATH, yr_path).click()

    # Select Colleges 
    WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[@id="subagencyGroupHeading"]/h4/a')))
    driver.find_element(By.CSS_SELECTOR, "#subagencyGroupHeading a").click()
    driver.find_element(By.CSS_SELECTOR, "#subagencyGroup .form-control").click()

    # WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, '//*[@id="subagencyGroupHeading"]/h4/a')))
    driver.find_element(By.XPATH, '//*[@id="subagencyGroup"]/div/input').send_keys("College")

    time.sleep(5)

    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="subagencyGroup"]/ul/li[1]/span[1]')))

    elem = driver.find_elements_by_xpath('//*[@id="subagencyGroup"]')
    all_li = elem.find_elements_by_tag_name("li")

    # Select SUNY and CUNY 
    for el in tqdm(all_li):

        # link_to_use = '//*[@id="subagencyGroup"]/ul/li[' + str(box) + ']/span[1]'
        WebDriverWait(driver, 10).until(EC.presence_of_element_located(el)
        el.click()
        time.sleep(1.5)

        #for i in tqdm(range(1,2500)):
        while not check_for_break_condition(driver):

            try: 
                WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, '//*[@id="data_loader"]')))
                driver.execute_script("stnyEvent.getNextPage();")
            except:
                captureError(driver)

        el.click()

Answer 1

試試這個。 看起來您可以通過發布請求獲得它。

import requests
import pandas as pd

s = requests.Session()
s.get('https://www.seethroughny.net/')

cookies = s.cookies.get_dict()
cookieStr = ''
for k,v in cookies.items():
    cookieStr += f'{k}={v};'

url = 'https://www.seethroughny.net/tools/required/reports/payroll'
headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36',
        'cookie':cookieStr,
        'accept': 'application/json, text/javascript, */*; q=0.01',
        'referer': 'https://www.seethroughny.net/payrolls'}
payload = {'action':'get'}


final_df = pd.DataFrame()
for year in [2008, 2009, 2010]:
    print(year)
    success = True
    page = 0
    while success == True:
        formData = {
        'PayYear[]': f'{year}',
        'SortBy': 'YTDPay DESC',
        'current_page': f'{page}',
        'result_id': '0',
        'url': '/tools/required/reports/payroll?action=get',
        'nav_request': '0'}

        try:
            response = s.post(url, headers=headers, data=formData, params=payload, timeout=10).json()['html']
            success = True
            df = pd.read_html('<table>' + response)[0]
            final_df = final_df.append(df, sort=False).reset_index(drop=True)
            print(f'Page: {page}')
            page+=1
        except:
            success = False

Output：

print(final_df.head(5).to_string())
    0                 1                                          2           3                                           4
0 NaN     Hohn, David C  Roswell Park Cancer Institute Corporation  $1,519,572   Roswell Park Cancer Institute Corporation
1 NaN   Israel, Michael                      Municipal Authorities  $1,219,418  Westchester County Health Care Corporation
2 NaN   Onesti, Stephen                                       SUNY    $947,463              Downstate Medical Center(Hosp)
3 NaN        Ko, Wilson                                       SUNY    $817,116              Downstate Medical Center(Hosp)
4 NaN  Marmur, Jonathan                                       SUNY    $768,594              Downstate Medical Center(Hosp)
...