Pycharm: Selenium with Python: Unable to find web elements using headless chrome

Question

When I run the below code with headless chrome I'm facing issues while identifying the elements. The same code runs just runs fine by commenting the below lines with head full mode.

# chrome_options.add_argument('--headless')

# chrome_options.add_argument('--disable-gpu')

Test Details:

Operating system: Windows10

ChromeDriver: 75.0.3770.8

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test@email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()

Output:

"C:\Program Files (x86)\Python37-32\python.exe" C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py

Checking for win32 chromedriver:75.0.3770.8 in cache
Driver found in C:\Users\vishr\.wdm\chromedriver\75.0.3770.8\win32/chromedriver.exe
Home | Let's Kode It
Traceback (most recent call last):
  File "C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py", line 15, in <module>
    wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
  File "C:\Users\vishr\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 


Process finished with exit code 1

Answer 1

for headless browser you have to set the window size to fire on event.Because headless browser can't recognise where to click without window size.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('window-size=1920x1080')

driver = webdriver.Chrome(executable_path='path/to/chrome driver',options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test@email.com")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()

Printed output on console on headless mode.

Home | Let's Kode It
Let's Kode It

Answer 2

You're clicking a button that loads another page. You have to wait for the other page to load. This is tricky to get perfect.

The easiest solution:

sleep(3)

The better solution using implicit waiting:

driver.implicitly_wait(3)

An even better solution using explicit waiting:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.ID, "user_email")))

An even better better solution would be to catch exceptions, and layer all these solutions.

Answer 3

VISHVAMBRUTHJAVAGALTHIMMEGOWDA,

I tried your code and was getting the same exception, First I thought that username was in Frame or iframe but it is not.

Then I tried to introduce webdriver wait and it worked just fine :

Code :

wait = WebDriverWait(driver,10)

driver.maximize_window()

driver.get("https://learn.letskodeit.com/")

print(driver.title)

driver.find_element_by_xpath("//a[contains(text(),'Login')]").click()

wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test@email.com")

#driver.find_element_by_id("user_email").

driver.find_element_by_id("user_password").send_keys("abcabc")

driver.find_element_by_name("commit").click()

print(driver.title)  

Remember to imports these :

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 

EDIT 1 :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')  # Last I checked this was necessary.
driver = webdriver.Chrome("C:/Users/XXXX/Downloads/BrowserDriver/chromedriver_win32/chromedriver.exe", chrome_options=options)
wait = WebDriverWait(driver,10)

driver.get("https://learn.letskodeit.com/")

print(driver.title)


wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()

wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("test@email.com")

wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")

wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()

print(driver.title)

Answer 4

To click() on the element with text as Login you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following solutions:

• Using PARTIAL_LINK_TEXT :

   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Login"))).click() 

• Using CSS_SELECTOR :

   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.navbar-link.fedora-navbar-link[href='/sign_in']"))).click() 

• Using XPATH :

   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='navbar-link fedora-navbar-link' and @href='/sign_in']"))).click() 

• Note : You have to add the following imports :

   from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC