如何用循环和 selenium 解决这个问题？

Question

So I'm trying to extract the URL of the first organic search result of a certain word on Google using Selenium Web Driver and Python. It goes like this:

from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
driver = webdriver.Chrome(chrome_options=chrome_options)

sample =[]
news = ["a", "b", "c"]

for item in range(len(news)):
    driver.get('https://www.google.com')
    driver.find_element_by_xpath('search_bar_xpath').send_keys(news[item+1])
    driver.find_element_by_xpath('search_button_xpath').click()
    domain = driver.find_element_by_xpath('first_result_url_xpath').text
    sample.append(domain)
    driver.close()

Well, it doesn't loop. Can you guys kindly help me fix it. And sometimes the first_result_url_xpath change to another name. How can I tell it to search for the other and not worry about the first try?

Answer 1

The problem is, that you close the driver in the first cycle of your loop. You should get it out from the loop:

for item in range(len(news)):
    driver.get('https://www.google.com')
    driver.find_element_by_xpath(search_bar_xpath).send_keys(news[item+1])
    driver.find_element_by_xpath(search_button_xpath).click()
    domain = driver.find_element_by_xpath(first_result_url_xpath).text
    sample.append(domain)
driver.close()

And you can get multiple result by using: driver.findElements(By.xpath(first_result_url_xpath)) and then iterate on it to find what you are searching for.