timeout exception error on using headless chrome webdriver

Question

My selenium code is working without headless mode but when I try to do with headless mode, it raise TimeOutException error. I am using python selenium chrome webdriver. I am testing this on my local machine at address http://127.0.0.1:5000/ I am trying to log in Instagram with selenium but it's not working in headless mode. I tries to resize the window (maximizing, manually setting the size, resize_to) but none of them working.

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("—disable-gpu")
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_driver = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(chrome_driver,chrome_options=chrome_options)
driver.set_window_size(1382, 744)

I am using this below helper function to wait for elements

def waiting(xpath):
    element = WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.XPATH, xpath))
    )
    return element

My code to login in Instagram is below:

driver.get("https://www.instagram.com")
username = waiting("/html/body/div[1]/section/main/article/div[2]/div[1]/div/form/div/div[1]/div/label/input")
username.send_keys("abcdeff")
password = waiting("/html/body/div[1]/section/main/article/div[2]/div[1]/div/form/div/div[2]/div/label/input")
password.send_keys("password")
driver.find_element_by_xpath("/html/body/div[1]/section/main/article/div[2]/div[1]/div/form/div/div[3]/button").click()

I am getting this error:

Traceback (most recent call last):
  File "C:\Users\madhav\AppData\Local\Programs\Python\Python39\lib\site-packages\flask\app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\madhav\AppData\Local\Programs\Python\Python39\lib\site-packages\flask\app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\madhav\AppData\Local\Programs\Python\Python39\lib\site-packages\flask\app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\madhav\AppData\Local\Programs\Python\Python39\lib\site-packages\flask\app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "C:\Users\madhav\Desktop\Fprojects\insta-sam\temp.py", line 47, in login
    username = waiting("/html/body/div[1]/section/main/article/div[2]/div[1]/div/form/div/div[1]/div/label/input")
  File "C:\Users\madhav\Desktop\Fprojects\insta-sam\temp.py", line 30, in waiting
    element = WebDriverWait(driver, 20).until(
  File "C:\Users\madhav\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Answer 1

I have tried the below code and it is working for me. Please give it a try and if it solves your problem then please mark it as answer. Remember to change yourUserName and YourPassword in the code.

from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")

driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 30)
action = ActionChains(driver)

driver.get("https://www.instagram.com/")

wait.until(EC.presence_of_element_located((By.NAME, "username"))).send_keys("yourUserName")
wait.until(EC.presence_of_element_located((By.NAME, "password"))).send_keys("YourPassword")
wait.until(EC.presence_of_element_located((By.XPATH, "//div[text()='Log In']/parent::button"))).click()

print(driver.current_url)

Note - Instagram is detecting that you are using selenium in headless mode. To avoid that you can use the fake useragent like options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36") .

Hope it will resolve your problem. Thank you.!

Answer 2

You are getting an error at step username . I generally don't prefer using "WebDriverWait()....until(.....)". I always use:

import time
...
...
time.sleep(10)
...
...

This always solves my problem, and is efficient as well, As Swaroop has mentioned, you should avoid using the complete xpath as far as possible. The code may breakdown.