Selenium Unable to locate element only when using headless chrome (Python)

Question

I just started learning Selenium and need to verify a login web-page using a jenkins machine in the cloud, which doesn't have a GUI. I managed to run the script successfully on my system which has a UI. However when I modified the script to run headless, it fails saying unable to locate element. My script is as follows:

#!/usr/bin/env python3

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
import time
import argparse


chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1120, 550')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--allow-running-insecure-content')

driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=chrome_options)
driver.implicitly_wait(5)

lhip = '13.14.15.16'
user = 'username'
paswd = 'password'


parser = argparse.ArgumentParser()

parser.add_argument('-i', '--lh_ip',    type=str, metavar='', default=lhip,     help='Public IP of VM' )
parser.add_argument('-u', '--usr',      type=str, metavar='', default=user,     help='Username for VM')
parser.add_argument('-p', '--pwd',      type=str, metavar='', default=paswd,    help='Password for VM')

args = parser.parse_args()


lh_url = 'https://' + args.lh_ip + '/login/'
driver.get(lh_url)
try:
    if driver.title == 'Privacy error':
        driver.find_element_by_id('details-button').click()
        driver.find_element_by_id('proceed-link').click()
except:
    pass

driver.find_element_by_id('username').send_keys(args.usr)
driver.find_element_by_id('password').send_keys(args.pwd)
driver.find_element_by_id('login-btn').click()
driver.implicitly_wait(10)
try:
    if driver.find_element_by_tag_name('span'):
        print('Login Failed')
except:
    print('Login Successful')
driver.close()

The python script works fine on my system when used without the chrome_options. However upon adding them to run in headless mode, it fails with the following output:

[WDM] - Current google-chrome version is 85.0.4183
[WDM] - Get LATEST driver version for 85.0.4183
[WDM] - Driver [/home/ramesh/.wdm/drivers/chromedriver/linux64/85.0.4183.87/chromedriver] found in cache
Traceback (most recent call last):
  File "/home/ramesh/practice_python/test_headless.py", line 44, in <module>
    driver.find_element_by_id('username').send_keys(args.usr)
  File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/ramesh/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="username"]"}
  (Session info: headless chrome=85.0.4183.121)

Since I have about one day's learning of Selenium, I may be doing something rather silly, so would be very grateful if someone showed me what I've done wrong. I've googled a lot and tried many things but none worked. Also why is it saying "css selector" when I have only used id for username?

Answer 1

If the script is working perfectly fine without headless mode, probably there is issue with the window size. Along with specifying --no-sandbox option, try changing the window size passed to the webdriver

chrome_options.add_argument('--window-size=1920,1080')

This window size worked in my case.

Even if this dosen't work you might need to add wait timers as answered before as rendering in headless mode works in a different way as compared to a browser in UI mode.

Ref for rendering in headless mode - https://www.toolsqa.com/selenium-webdriver/selenium-headless-browser-testing/

Answer 2

I would refactor code in a way to wait until elements will be present on a web page:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.ID, 'username'))).send_keys(args.usr)
WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.ID,'password'))).send_keys(args.pwd)
WebDriverWait(wd, 10).until(EC.presence_of_element_located((By.ID, 'login-btn'))).click()

Generally usage of WebDriverWait in combination with some condition should be preferred to implicit waits or time.sleep() . Here is explained in details why.

Other thing to double check are whether elements have the IDs used for search and that these elements aren't located within an iframe.

Answer 3

I had the same issue, it was initially working, though after an update from a website that we were using Selenium on, it stopped working in headless mode, though kept on working in non headless. After 2 days of researching the deepest and darkest depths of the web and a lot of trial and error, finally found what the issue was.

I tried all the methods outlined on the web and more, though nothing worked until I found this.

In headless chrome mode, the user agent is something like this: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome /60.0.3112.50 Safari/537.36

The service provider updated their code to identify the HeadlessChrome part and would cause the tab to crash and in turn, destroying the Selenium user session.

Which lead to the issue firing above in one of the exceptions.

To solve this, I used a plugin called fake_headers ( https://github.com/diwu1989/Fake-Headers ):

from fake_headers import Headers

header = Headers(
    browser="chrome",  # Generate only Chrome UA
    os="win",  # Generate only Windows platform
    headers=False # generate misc headers
)
customUserAgent = header.generate()['User-Agent']

options.add_argument(f"user-agent={customUserAgent}")

Though this was only half the solution as I only wanted Windows and Chrome headers and the fake_headers module has not not include the latest Chrome browsers and has a lot of older versions of Chrome in the list as can be seen in this file https://github.com/diwu1989/Fake-Headers/blob/master/fake_headers/browsers.py . The particular site I was running Selenium had certain features that only worked on newer versions of Chrome, so when an older version of Chrome was passed through the user-agent header, the certain features on the site would actually stop working. So I needed to update the browsers.py file in the fake_headers module to include only the versions of Chrome that I wanted to include. So I deleted all the older versions of Chrome and created a select list of versions (each one individually tested to work on the site in question and deleted the ones that did not). Which ended up with the following list, which I could expand on though have not for the time being.

chrome_ver = [
    '90.0.4430', '84.0.4147', '85.0.4183', '85.0.4183', '87.0.4280', '86.0.4240', '88.0.4324', '89.0.4389', '92.0.4515', '91.0.4472', '93.0.4577', '93.0.4577'
]

Hope this helps spare someone two days of stress and messing around.

Some more useful info on headless chrome detectability: https://intoli.com/blog/making-chrome-headless-undetectable/

Answer 4

I came across similar situations too. And I've tried a lot of solutions online such as specifying the resolution, but nothing worked until this:

self.chrome_options.add_argument('user-agent="MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"')

So it seems that you need to add UA to chrome options so that the selenium driver wouldn't crash in headless mode.