python/Selenium --> find_elements_by_xpath method not finding all elements

Question

I try to extract 10 elements from a website using find_elements_by_xpath but it always extracts the first four elements.

Here is a screenshot of what i want to extract (lines marked with a red dot and it stops at the line marked with a red cross):

and here is my python code:

################################################################################
#   Déclaration et initialisation des variables globales
################################################################################

currentDirectory = os.getcwd()
path_to_website = "URL" #changed for stackoverflow question
path_to_chrome_profile = "path_to_chrome_profile" #changed for stackoverflow question

xpath_suggestions_text = "//li[@class='search-result search-result__occluded-item ember-view']"

################################################################################
#   Navigation sur google Chrome
################################################################################

# lancement de chrome avec le profil par défaut
options_ch = webdriver.ChromeOptions()
options_ch.add_argument("user-data-dir="+path_to_chrome_profile)

browser_chrome = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options_ch)

# on va sur la page réseau de linkedIn
browser_chrome.get(path_to_website )

list_elements = browser_chrome.find_elements_by_xpath(xpath_suggestions_text)


print(len(list_elements )) #prints 4 instead of 10


for profils in list_elements :
    print(element.text)
    print("////////")



browser_chrome.quit()

I tried many things but nothing worked.. any clue? (i'm a beginner in web-scraping, please be indulgent:x )

Thank you all

AJT

Answer 1

You may want to try class selector instead of xpath.

browser.find_elements_by_class('search-result search-result__occluded-item ember-view')

if doesn't work, try to go from parent to child.

div > ul

Answer 2

Ok I just had to scroll down to load the web page and therefore all elements. I was tricked by chrome HTML inspector that directly shows all elements of the webpage.

I think that @JaSON had the right answer