How to click buttons with Python Selenium webdriver on this website when the element cannot be found?

Question

I'm very new to Selenium with Python and I'm simply trying to click 2 consecutive buttons on this website: https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28

I want to click the settings icon in the top right corner and then click the celsius option to convert temperatures on the site.

I can achieve this whilst debugging the code and doing the steps one at a time, but when I run the code normally, it is unable to find the second element with the error:

NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="wuSettings-quick"]/div/a[2]"}
  (Session info: chrome=85.0.4183.83)

I have tried identifying the elements by id and xpath as follows:

Button 1: driver.find_element_by_id('wuSettings').click() Button 2: driver.find_element_by_xpath('//*[@id="wuSettings-quick"]/div/a[2]').click()

Hoping it's an easy fix. All help appreciated. Thanks.

Code below:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28")
driver.implicitly_wait(15)

try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, 'wuSettings'))
    )
    element.click()
    
    # element = WebDriverWait(driver, 10).until(
    #     EC.presence_of_element_located((By.XPATH, '//*[@id="wuSettings-quick"]/div/a[2]'))
    # )
    # element.click()
except:
    driver.quit()

Answer 1

如果我替换此行，您的代码工作正常： driver.implicitly_wait(15)与driver.implicitly_wait(15) time.sleep(5)

Answer 2

When I need to await for an element to be clickable, I usually have the best luck with something like this

can_click=False
while not can_click:
    try:
        element=driver.find_element_by_id('wuSettings')
        element.click()
        # note that if the issue is that the element is not visible in
        # the viewport, you may need to do 
        # driver.execute_script("arguments[0].click();", element) 
        # instead
        can_click=True
    except:
        pass

basically it should throw an exception until you can click on the element you want and once it does the loop terminates.

Answer 3

I tried different things, but it looks like you have to agree cookies use before you can click on another button.

I tried Matt's solution but I'm pretty sure that the commercial loading on top of the page screws up the first click on the settings button.

This is working fine for me :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28")

cookies = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//button[@id="truste-consent-button"][@type="button"]'))
)
cookies.click()

time.sleep(5)

element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//button[@id="wuSettings"][@class="wu-settings"]'))
)
element.click()

element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//*[@id="wuSettings-quick"]/div/a[2]'))
)
element.click()

Answer 4

First of all you should not use both Implicit and explicit wait together. A cocktail of it could cause an unpredictable wait times. As you are using explicit wait for both buttons, it should be more than sufficient.

Also use of time.sleep might work in this case but its never been a good way to wait in test script as it makes test brittle and unpredictable.

As you can see even after page load few ajax components (Record for day passed in url is loading) are still loading and even though setting element is present its not giving option to change temp, unit record is fully loaded. So explicit / implicit wait on setting icon is not working.

Try below code and it should work.

try: 
    driver.get("https://www.wunderground.com/history/daily/gb/manchester/EGCC/date/2017-8-28")
    WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "//div[@class='legend-def temperature']"))) # To insure record for the day is loaded 
    ele = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, "wuSettings")))
    driver.execute_script("arguments[0].scrollIntoView();", ele)
    ele.click() #Scroll to Setting icon
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[@title='Switch to Metric']"))).click()
except:
    driver.quit()