[英]python/Selenium --> find_elements_by_xpath method not finding all elements
I try to extract 10 elements from a website using find_elements_by_xpath but it always extracts the first four elements.我尝试使用 find_elements_by_xpath 从网站中提取 10 个元素,但它总是提取前四个元素。
Here is a screenshot of what i want to extract (lines marked with a red dot and it stops at the line marked with a red cross):这是我要提取的内容的屏幕截图(标有红点的行,它停在标有红十字的行处):
and here is my python code:这是我的 python 代码:
################################################################################
# Déclaration et initialisation des variables globales
################################################################################
currentDirectory = os.getcwd()
path_to_website = "URL" #changed for stackoverflow question
path_to_chrome_profile = "path_to_chrome_profile" #changed for stackoverflow question
xpath_suggestions_text = "//li[@class='search-result search-result__occluded-item ember-view']"
################################################################################
# Navigation sur google Chrome
################################################################################
# lancement de chrome avec le profil par défaut
options_ch = webdriver.ChromeOptions()
options_ch.add_argument("user-data-dir="+path_to_chrome_profile)
browser_chrome = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options_ch)
# on va sur la page réseau de linkedIn
browser_chrome.get(path_to_website )
list_elements = browser_chrome.find_elements_by_xpath(xpath_suggestions_text)
print(len(list_elements )) #prints 4 instead of 10
for profils in list_elements :
print(element.text)
print("////////")
browser_chrome.quit()
I tried many things but nothing worked.. any clue?我尝试了很多东西,但没有任何效果..有什么线索吗? (i'm a beginner in web-scraping, please be indulgent:x )
(我是网络抓取的初学者,请放纵:x)
Thank you all谢谢你们
AJT AJT
You may want to try class selector instead of xpath.您可能想尝试 class 选择器而不是 xpath。
browser.find_elements_by_class('search-result search-result__occluded-item ember-view')
if doesn't work, try to go from parent to child.如果不起作用,请尝试从父母到孩子 go。
div > ul
div > ul
Ok I just had to scroll down to load the web page and therefore all elements.好的,我只需要向下滚动即可加载 web 页面以及所有元素。 I was tricked by chrome HTML inspector that directly shows all elements of the webpage.
我被直接显示网页所有元素的 chrome HTML 检查器欺骗了。
I think that @JaSON had the right answer我认为@JaSON 有正确的答案
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.