Can't locate element on Pop-Up Window using Selenium on Python

Question

I want to scrape some elements from the Duden webpage with this url: https://www.duden.de/rechtschreibung/aussuchen . When I look up the page manually no pop-up occures but when I use selenium on python this occures: image of pop up

I already tried a lot of things like blocking pop ups in general, or trying to click on the accept button. All of that is not working.

I tried to find an element of the frame and print a statement then to see whether it can find the elements but that is also not working.

Has anyone an idea why it is like that or what I could try more?

These are a few things I tried:

• For blocking:

   def getAllWordForms(word): options = Options() profile = webdriver.FirefoxProfile() profile.set_preference("dom.disable_open_during_load", False) driver = webdriver.Firefox(firefox_profile=profile,options=options, executable_path=os.path.join(driver_location, 'geckodriver')) main_url = 'https://www.duden.de/rechtschreibung/' word_url = main_url + '{}'.format(word) driver.get(word_url)

• to see if it can find an element in the pop up frame:

   def getAllWordForms(word): options = Options() driver = webdriver.Firefox(options=options, executable_path=os.path.join(driver_location, 'geckodriver')) main_url = 'https://www.duden.de/rechtschreibung/' word_url = main_url + '{}'.format(word) driver.get(word_url) driver.implicitly_wait(10) driver.switch_to.frame(1) if driver.find_elements_by_class_name('message-button'): print('yes')

• to click the button:

   def getAllWordForms(word): options = Options() options.headless = False driver = webdriver.Firefox(options=options, executable_path=os.path.join(driver_location, 'geckodriver')) main_url = 'https://www.duden.de/rechtschreibung/' word_url = main_url + '{}'.format(word) driver.get(word_url) driver.implicitly_wait(10) driver.switch_to.frame(1) button = driver.find_element_by_xpath("//button[@aria-label='AKZEPTIEREN']") button.click() driver.switch_to.default_content()

I tried out various combinations, but it never works.

The elements of the page are structred like this: structure of page_1 structure of page_2

Hope I could explain it correct and that maybe someone could help me.

Answer 1

Every time you launch your webdriver you're using a new temporary profile. That profile has no cookies therefore it's seen by the site as a new user an needs to accept the cookie message.

I had a look at your site and to close the message you need to switch iframe. You were close with your solution, it might just be it needed a different method of selecting the frame...

This code works for me:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.duden.de/rechtschreibung/aussuchen")

iframe = driver.find_element_by_xpath("//iframe[contains(@id,'sp_message_iframe')]")
driver.switch_to.frame(iframe)
cookieAccpet = WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='AKZEPTIEREN']")))
cookieAccpet.click()

driver.switch_to.default_content()

Remember to switch back to the default frame at the end with driver.switch_to.default_content() , then you can continue your script.