使用 Selenium/Python 加载所有元素 - 加载更多

Question

我正在尝试刮掉位于https://www.fragrantica.com/search/的所有香水

网站上有近 73,367 种香水，我想加载所有这些香水。 问题是该网站显示了 30 种香水，然后您需要单击“显示更多结果”，这将显示另外 30 种香水等等。所以基本上我们必须按显示更多按钮几乎 2444 次才能到达页面末尾并且装满所有香水。 到目前为止，这是我的代码：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
import time


options = Options()
options.add_argument("--profile-directory=Default")
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36")


driver = webdriver.Chrome(service = Service(executable_path='C:/Users/armon/Downloads/chromedriver_win32/chromedriver.exe'), options=options)
url = 'https://www.fragrantica.com/'
driver.get(url)

time.sleep(3)

perfumes_btn = driver.find_element(by=By.XPATH, value = '//*[@id="offCanvasLeft"]/ul/li[5]/a')
perfumes_btn.click()

search_btn = WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[2]/div[2]/div[2]/ul/li[5]/ul/li[1]/a')))
search_btn.click()


load_more_btn = '//*[@id="main-content"]/div[1]/div[1]/div/div/div/div[2]/div[1]/div/div[3]/div/div/div/div/div/button'
loadingButton = WebDriverWait(driver,70).until(EC.element_to_be_clickable((By.XPATH,load_more_btn)))

maxPerfumes = 73346;
i = 0

while loadingButton:
    driver.execute_script("arguments[0].click();", loadingButton)
    i = i+1
    print(i)    
    loadingButton = WebDriverWait(driver,70).until(EC.element_to_be_clickable((By.XPATH,load_more_btn)))
    loadElems = driver.find_elements(by = By.XPATH, value = '//*[@id="main-content"]/div[1]/div[1]/div/div/div/div[2]/div[1]/div/div[3]/div/div/div/span/div[1]')
    if len(loadElems)>0:
        loadingButton = WebDriverWait(driver,70).until(EC.element_to_be_clickable((By.XPATH,load_more_btn)))
        PerfumesLoaded = len(driver.find_elements(by = By.XPATH, value = '//*[@id="main-content"]/div[1]/div[1]/div/div/div/div[2]/div[1]/div/div[3]/div/div/div/span/div[1]'))
    else:
        print("Loaded all the tires")
        break
    if PerfumesLoaded >= maxPerfumes:
        print (PerfumesLoaded + " are loaded successfully.")
        break

问题是每次我运行它时，它最多只能循环 34 次，然后停止并抛出错误：

TimeoutException                          Traceback (most recent call last)
c:\Users\armon\OneDrive\Desktop\OLD\Data Analytics\Portfolio Projects\Jolse Project\Jolse Scraping Notebook.ipynb Cell 1' in <cell line: 36>()
     41 loadElems = driver.find_elements(by = By.XPATH, value = '//*[@id="main-content"]/div[1]/div[1]/div/div/div/div[2]/div[1]/div/div[3]/div/div/div/span/div[1]')
     42 if len(loadElems)>0:
---> 43     loadingButton = WebDriverWait(driver,70).until(EC.element_to_be_clickable((By.XPATH,load_more_btn)))
     44     PerfumesLoaded = len(driver.find_elements(by = By.XPATH, value = '//*[@id="main-content"]/div[1]/div[1]/div/div/div/div[2]/div[1]/div/div[3]/div/div/div/span/div[1]'))
     45 else:

File c:\Users\armon\anaconda3\envs\armoniaenv\lib\site-packages\selenium\webdriver\support\wait.py:87, in WebDriverWait.until(self, method, message)
     85     if time.monotonic() > end_time:
     86         break
---> 87 raise TimeoutException(message, screen, stacktrace)

TimeoutException: Message: 

我在循环中的 print(i) 只是为了知道它做了多少个循环。关于如何调整它的任何建议？ 或者如果我做错了什么？

我的目标是单击加载更多，直到我不能再加载，然后我可以访问页面上的所有香水。 TIA

Answer 1

我在查找元素时使用 try-except 来解决它。 您还可以在 while 循环结束后仅更改一次代码以获取结果

try:
    loadingButton = WebDriverWait(driver,70).until(EC.element_to_be_clickable((By.XPATH,load_more_btn)))

except Exception as e:
     print("cant locate element: %s" %e)
     loadingButton = False

#我的代码来解决这个问题

more = True 
while more:
        
    more = my_custom_function("//XX")
    #my_custom_function try to get element by Xpath
    #and also click it, on exception it will return False
    
    if more:
        #Need sleep cause website change when click load more
        # python code can run faster than website loads, can do sleep inside my_custom_function also
        sleep(1)