Selenium 驱动程序不返回所有 web 元素，即使它们是相同的 class

Question

I'm trying to make a web scraper for a webpage using selenium.

the webpage is https://www.supersport.hr/sport/dan/0/sport/1

I am able to get most of the web elements I'm trying to get but my script doesn't return all of them even though they are the same class.

page example:

In this case my script returns all of the above divs, but cuts off at "SRL GRČKA 1." and doesn't get any leagues below.

the HTML markup for each of these leagues are the same:

I'm getting a list of these elements in python like this:

football_leagues_elements = driver.find_elements_by_css_selector("div.sportska-liga.nogomet")

I've also tried with this code but it returns the same result:

football_leagues_elements = driver.find_elements_by_xpath("//div[contains(@class, 'sportska-liga-wrap')]//div[contains(@class, 'nogomet')]")

I think all the leagues are loaded to the page at the same time.

My question is why are some divs not included in the webelement list?

Any help is welcome.

Answer 1

The Website is NOT accessible here.However try Induce Explicit wait and wait for all element visible.

football_leagues_elements=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.sportska-liga.nogomet")))

You need to import following libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

---

If above doesn't work then try to scroll the page first and then check for elements.

#Scroll bottom of the page.
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
#Get the elements.
football_leagues_elements=WebDriverWait(driver,20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"div.sportska-liga.nogomet")))