Selenium find_elements_by_id() doesn't return all elements

Question

I am trying to find all elements of a certain kind on a webpage with Selenium (python). For simplicity, let's say their id is elem_id . I am using the following code snippet to do so:

all_elements = driver.find_elements_by_id("elem_id")
print(str(len(all_elements)))

I know that there are ~3000 of this kind of element on the webpage in question, but whenever I print the length of all_elements , it always prints 1000 .

It definitely finds the right kind of element (I checked), but somehow it doesn't find all of them at once. It also selects the 1000 elements at random, meaning that it neither picks the first 1000 or the last 1000 exclusively. I tried finding out whether there's a cap on how many elements Selenium can find, but there doesn't appear to be a max amount of 1000.

Does anyone know why Selenium only finds 1000 elements at a time? What am I doing wrong? Thank you very much!

Answer 1

Fundamentally, you are seeing the correct behavior. Though you are aware there are almost ~3000 of this particular type of element within the webpage, but:

• All those elements with id as elem_id may not be visible within the Viewport

You can find a relevant detailed discussion in How does Selenium click on elements that are 50% on screen and 50% not on screen?

• Some of those elements may be within <iframe> / <frame> tags and WebDriver instance may not have the visibility of those elements from the Top Level View .

You can find a relevant detailed discussion in Ways to deal with #document under iframe

• Some of those elements may remain invisible through Lazy Loading

You can find a relevant detailed discussion in How to click on Load More button within Google Trends and print all the titles through Selenium and Python

Hence you see only ~1000 of those elements out of ~3000 odd elements.