使用 Selenium Web 驱动程序调用时出错：driver.get(linked_url.get_atrribute(“href”))，过时的元素引用

Question

I am working with selenium in python. I am extracting the links from google search results which I have done successfully lower now I am trying to navigate to these one by one using a for loop and a driver.get() method:

search_gog=driver.find_element_by_name('q')
search_gog.send_keys(parameters.search_gog)
sleep(0.5)
search_gog.send_keys(Keys.RETURN)
sleep(3)linkedin_urls = driver.find_elements_by_xpath("//div[@class='g']//div[@class='r']/a[contains(@href, 'https://www.linkedin.com')]") 

for linkedin_url in linkedin_urls:
  
    driver.get(linkedin_url.get_attribute("href")) 
    

I have successfully extracted the urls. I verified this with this code:

linkedin_urls = driver.find_elements_by_xpath("//div[@class='g']//div[@class='r']/a[contains(@href, 'https://www.linkedin.com')]") 

for linkedin_url in linkedin_urls:
  
  print(linkedin_url)

this returns the list of urls that I have extracted. However when running the first code snippet I get the following error:

Traceback (most recent call last):
  File "app2.py", line 45, in <module>
    driver.get(linkedin_url.get_attribute("href")) #linkedin_url.get_attribute("href")
  File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webelement.py", line 139, in get_attribute
    attributeValue = self.parent.execute_script(
  File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 634, in execute_script
    return self.execute(command, {
  File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=85.0.4183.102)

from my understanding the error that the stale element is not in the page source is due to either the element, when called having been deleted or if the element is not longer in the DOM. NO element has been deleted, and I can't seem to find a reason why the DOM might have changed. Any ideas why this error is occurring?

Answer 1

You have probably been directed to the next page because of which you are getting StaleElementReferenceException exception because after coming back to the main page the elements which are extracted gets removed from temporary memory. The solution for this can be extracting elements in the loop so that everytime you are trying to access some element it gets extracted that time only.

Now over here the for loop will not work, you have to use while loop. A sample code be like:

while(elementPresent(locatorType, locator)):
    link = driver.find_element_by_xpath(locator)
    #take action what you want to 

Over here the issue you will face is how many times the iteration need to be done. For that you may make a function like:

def elementPresent(locatorType, locator):
#present = true
#not present = false
wait = WebDriverWait(driver, 20)
try:
    wait.until(EC.presence_of_element_located((locatorType, locator)))
    wait.until(EC.visibility_of_element_located((locatorType, locator)))
except Exception:
    return False
return True

This function is basically checking the presence and visibility of element given when calling the function so that the elements can be extracted easily and without any error.

This has worked in my case.

Answer 2

I avoided this error by doing the following

list_links = [link.get_attribute('href') for link in driver.find_elements_by_xpath("//div[@class='g']//div[@class='r']/a[contains(@href, 'https://www.linkedin.com')]")]

    for link in list_links:
        driver.get(link)