Getting tImeout error when I am using python selenium headless

Question

Without using the selenium headless the following code works fine. But Why I am getting error when I am using headless mode. Here is my code:-

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
import time


options = Options()
options.add_argument("--disable-notifications")
options.headless = True
options.add_argument("--window-size=1920,1080")
driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)

url = "https://www.justdial.com/Delhi/S-K-Premium-Par-Hari-Nagar/011PXX11-XX11-131128122154-B8G6_BZDET"
driver.get(url)


pop_up = WebDriverWait(driver, 30).until(
    EC.element_to_be_clickable((By.XPATH, '//*[@id="best_deal_detail_div"]/section/span')))
time.sleep(2.5)
pop_up.click()  # For disable pop-up



while True:
     try:
       element = WebDriverWait(driver, 20).until(
        EC.element_to_be_clickable((By.XPATH, "//span[text()='Load More Reviews..']")))
       element.click()

    except TimeoutException:
       break
    except:
       pass


print([span.text for span in driver.find_elements_by_css_selector('span.rName.lng_commn')])

Kindly give some suggestions or solution. Thank you.

Answer 1

The root cause of the error is clear. It's happening as a result of the script not being able to locate the element -that is held by an explicit-wait .

Given that you tried the answers mentioned in the comments then this could also be to do with the anomaly behaviour issue in non-headless and headless mode. There are situations in which a headless mode cannot detect elements by their id in which case pass in the full absolute-path of the DOM .

Eg In the following, pass full xpath and see if it solves the problem.

element = WebDriverWait(driver, 20).until(
        EC.element_to_be_clickable((By.XPATH, <full xpath>)))
       element.click()