使用带有 webdriver.find 函数的 python 过滤使用 selenium 进行网页抓取

Question

I'm filtering the fixed income products which appears in this website: https://yubb.com.br/investimentos/renda-fixa?investment_type=cdb&months=3&principal=10000000.0&sort_by=minimum_investment

Basically, the page have some cards and I wanna know for each page how many cards appears. For example, choosing cdb as type and 3 months, it shows 16 cards, but with another input of months or type of product, it may appear less cards.

by now, I know how many possible number of pages it will appear looking at "investmentCardContainer__footer", which is a class but the number of cards looks like it shows as style and I dont know how to find it using selenium webdriver.find function.

Here's a tip for what i'm looking for:

https://imgur.com/a/8B5TrMe

The idea it's to get this number of cards and use it in a loop to get the cards information aggregated in a vector.

    vetor = ["cdb","lca","lci"]
    dataset_boxes =[]
    now = time.time()
    for i in vetor:
      options = Options()
      options.add_argument('--headless')
      url = 'https://yubb.com.br/investimentos/renda-fixa?investment_type={}&months=12\
        &principal=1000000.0&sort_by=net_return'.format(i)
      driver = webdriver.Chrome("C:\\Users\\yourpath\\Desktop\\PYTHON\\chromedriver.exe",options=options)
      driver.get(url)
      time.sleep(1)
      num_pages = driver.find_element_by_class_name("investmentCardContainer__footer").text
      list_pages = Convert(num_pages)
      last_page  = int(list_pages[len(list_pages)-3])
      driver.quit()
        for j in range(1,last_page+1):
          url2 = 'https://yubb.com.br/investimentos/renda-fixa?collection_page={}&investment_type={}&months=12\
            &principal=1000000.0&sort_by=net_return'.format(j,i)
          driver = webdriver.Chrome("C:\\Users\\yourpath\\Desktop\\PYTHON\\chromedriver.exe",options=options)
          driver.get(url2)
          num_boxes  = driver.find_element_by_class_name("investmentCardContainer__body").text
          list_boxes = Convert(num_boxes)
          dataset_boxes.append(list_boxes)
          driver.quit()
    print('idk')
    later = time.time()
    difference = int(later - now)
    print('Processo finalizado em {} segundos.'.format(difference)) 

Answer 1

Use WebDriverWait and following xpath to get the no of pages count.

print(WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,'(//span[@class="page"]//a)[last()]'))).text)

You need to have following imports to execute above code.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

For this link : https://yubb.com.br/investimentos/renda-fixa?investment_type=cdb&months=3&principal=10000000.0&sort_by=minimum_investment

It should return : 8