python selenium submit works interactively but not in a script, not even with time.sleep

Question

I'm trying to login to a site from a python script using the selenium webdriver in order to download some data that is only available to registered users.

I have the following code:

browser = webdriver.Firefox()
browser.get('https://shop.biogast.at/store15/customer/account/login')
emailElem = browser.find_element_by_id('email')
emailElem.send_keys('12345')
passwordElem = browser.find_element_by_id('pass')
passwordElem.send_keys('12345')
passwordElem.submit()

It works fine when the commands are typed to the python3 shell one by one. However, when I run the code as a script, the username and login are filled in correctly, but instead of logging in, the username and login fields are blanked. The script finishes and there is no error message.

According to what others have suggested in similar situations, this could be a timing issue, as the script runs much faster than when the commands are typed one by one. So I expanded the script to include some sleep values.

browser = webdriver.Firefox()
browser.get('https://shop.biogast.at/store15/customer/account/login')
time.sleep(15)
emailElem = browser.find_element_by_id('email')
emailElem.send_keys('12345')
time.sleep(15)
passwordElem = browser.find_element_by_id('pass')
passwordElem.send_keys('12345')
time.sleep(15)
passwordElem.submit()

Unfortunately, the result is still the same. The fields are blanked and the script finishes with no error. When I run the commands one by one, it works well even when the breaks between the commands are less than 15 seconds, so it really doesn't seem to be a timing problem.

Do you have any idea how should I find the cause? Thank you very much.

Answer 1

Use WebDriverWait to handle the element.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

browser = webdriver.Firefox()
wait = WebDriverWait(browser,40)
browser.get('https://shop.biogast.at/store15/customer/account/login')

emailElem = wait.until(EC.element_to_be_clickable((By.ID, 'email'))) #browser.find_element_by_id('email')
emailElem.send_keys('12345')

passwordElem = wait.until(EC.element_to_be_clickable((By.ID, 'pass'))) #browser.find_element_by_id('pass')
passwordElem.send_keys('12345')
passwordElem.submit()