python selenium, can't find elements from page_source while can find from browser

Question

I try find target element by xpath so that I can click on it. But can't find it when run code, although can find it by right-click option manually on chrome browser.

detail: I am using

driver.get('chrome://settings/clearBrowserData')

to get history pop-up from chrome, then wait element by selenium, and next action I try to click it by:

driver.find_element_by_css_selector('* /deep/ #clearBrowsingDataConfirm').click()

or by:

driver.find_element_by_xpath(r'//paper-button[@id="clearBrowsingDataConfirm"]').click()

both does not work

Could you tell solution by xpath if possible because I am more familiar with it. Or any other way to clear history on chrome, thank

Answer 1

Looking into Chrome Settings page source it looks like the button, you're looking for is hidden in the ShadowDOM

So you need to iterate down several levels of ShadowRoot

So the algorithm looks like:

1. Locate parent WebElement
2. Locate its shadow-root and cast it to the WebElement
3. Use WebElement.find_element() function to locate the next WebElement which is the parent for the ShadowRoot
4. Repeat steps 1-3 until you're in the same context with the element you want to interact with

Example code:

from selenium import webdriver

def expand_root_element(element):
    shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
    return shadow_root


driver = webdriver.Chrome("c:\\apps\\webdriver\\chromedriver.exe")
driver.maximize_window()
driver.get("chrome://settings/clearBrowserData")

settingsUi = driver.find_element_by_tag_name("settings-ui")
settingsUiShadowRoot = expand_root_element(settingsUi)

settingsMain = settingsUiShadowRoot.find_element_by_tag_name("settings-main")
settingsShadowRoot = expand_root_element(settingsMain)
settingsBasicPage = settingsShadowRoot.find_element_by_tag_name("settings-basic-page")
settingsBasicPageShadowroot = expand_root_element(settingsBasicPage)
settingsPrivacyPage = settingsBasicPageShadowroot.find_element_by_tag_name("settings-privacy-page")
settingsPrivacyShadowRoot = expand_root_element(settingsPrivacyPage)
settingsClearBrowsingDataDialog = settingsPrivacyShadowRoot.find_element_by_tag_name(
    "settings-clear-browsing-data-dialog")
settingsClearBrowsingDataDialogShadowRoot = expand_root_element(settingsClearBrowsingDataDialog)
settingsClearBrowsingDataDialogShadowRoot.find_element_by_id("clearBrowsingDataConfirm").click()

Answer 2

我通过这样做让它工作：

 driver.ExecuteScript("return document.querySelector('body > settings-ui').shadowRoot.querySelector('#main').shadowRoot.querySelector('settings-basic-page').shadowRoot.querySelector('#advancedPage > settings-section:nth-child(1) > settings-privacy-page').shadowRoot.querySelector('settings-clear-browsing-data-dialog').shadowRoot.querySelector('#clearBrowsingDataConfirm').click();");