Selenium find_element throwing exception even though the element exists

Question

My code:

soup = BeautifulSoup(driver.page_source,features="html.parser")
applications_domains = []

for card in soup.find_all("div", {"class":"ant-row"}):
    for url in card.find_all("a"):
    applications_domains.append(url.get("href"))

for aplications_domain in aplication_domains:
    try:
        WebDriverWait(driver,10).until(EC.presence_of_element_located((By.XPATH,"//a[@href='" + 
applications_domain + "']")))
        driver.find_element_by_xpath("//a[@href='" + applications_domain + "']").click()
    except:
        soup = BeautifulSoup(driver.page_source,features="html.parser")
        print(soup.find_all("a",{"href":applications_domain}))
        print(f"test error {applications_domain}")
        print("-----------------------")

I have an issue with find_element_by_xpath not finding the element even though it exists. I double checked using soup if indeed the element exists and it does as per output.

Output:

<a href="applications_domain"><b></b></a>
test error applications_domain

I have a loop that goes through each application domain (contains data from each href ) however, it finds and clicks on the a href element most of the time but does not for some and I have no idea why.

Here is the site html. There are many div id="application_name_list" and each contain different a href that I need to click through

<div class="ant-row" style="margin-left: -6px; margin-right: -6px;">
<div id="application_name_list" class="ant-col-8 dyff-home-app-search-result-item" style="padding-left: 6px; padding-right: 6px;">
    <a href="/dyfflaunch/domain/gco/app/di_data_customer_experience_conversation_processor/features">di_data_customer_experience_conversation_processor<b></b></a>
</div>
<div id="application_name_list" class="ant-col-8 dyff-home-app-search-result-item" style="padding-left: 6px; padding-right: 6px;">
    <a href="/dyfflaunch/domain/gco/app/di_kafka_configservice_agentqueuegroup_dim_v1-prod/features">di_kafka_configservice_agentqueuegroup_dim_v1-prod<b></b></a>
</div>
<div id="application_name_list" class="ant-col-8 dyff-home-app-search-result-item" style="padding-left: 6px; padding-right: 6px;">
    <a href="/dyfflaunch/domain/gco/app/di_kafka_configservice_phoneinventory_dim_v1-prod/features">di_kafka_configservice_phoneinventory_dim_v1-prod<b></b></a>
</div>
</div>
enter code here

Answer 1

This is one, pretty generic way of doing it:

a_tags=driver.find_elements_by_xpath("//div[@id='application_name_list']//a")

for a_tag in a_tags:
    a_tag.click()

If you have examples where this doesn't work, please add one to the question.

Answer 2

I would suggest use WebDriverWait() and wait for visibility_of_all_elements_located () and then use following css selector to click.

driver.get("url here")
WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".ant-row")))
for link in driver.find_elements_by_css_selector(".ant-row>#application_name_list>a[href]"):
    link.click()

---

If you want to use beautiful soup and selenium to do that then try this one.

driver.get("url here")
WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".ant-row")))

soup = BeautifulSoup(driver.page_source,features="html.parser")
applications_domains = []

for url in soup.select(".ant-row>#application_name_list>a[href]"):
    applications_domains.append(url['href'])

for applications_domain in applications_domains:
    try:
        WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//a[@href='" + applications_domain + "']")))
        driver.find_element_by_xpath("//a[@href='" + applications_domain + "']").click()
    except:
        soup = BeautifulSoup(driver.page_source,features="html.parser")
        print(soup.find_all("a",{"href":applications_domain}))
        print("test error {applications_domain}")
        print("-----------------------")

Answer 3

The issue was caused by overlapping and solved as per Solution Error message returned was selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point but due to my poor knowledge of error handling the error was not shown as expected. Thank you all for help!