Chrome + selenium + python works for headed chrome, does not work for headless run

Question

The following code works perfectly well with headed chrome, but fails with a timeout on the last line with headless chrome

import time                                                                               
import chromedriver_autoinstaller                                                         
from selenium import webdriver                                                            
from selenium.webdriver.common.by import By                                               
from selenium.webdriver.support.ui import WebDriverWait                                   
from selenium.webdriver.support import expected_conditions as EC                          
from selenium.webdriver.chrome.options import Options                                     
chromedriver_autoinstaller.install()                                                      
                                                                                          
chrome_options = Options()                                                                
chrome_options.add_argument("--headless")                                                 
                                                                                          
driver = webdriver.Chrome(options=chrome_options)                                         
                                                                                          
driver.get("http://redacted")                                          
                                                                                          
elem = WebDriverWait(driver, 60).until(                                                   
    EC.presence_of_element_located(                                                       
        (By.XPATH, "//div[contains(@class, 'new-book-rect')]")                            
    ))                                                                  

I checked the similar questions, but don't seem to apply. One indicates that you need to use xpath, rather than id, to lookup the element, but I am already doing so.

What is the reason of the failure, and is there a way to investigate what's going on when running headless?

Using the latest version of Chrome, and the driver is autodownloaded.

Answer 1

The Chromium developers recently added a 2nd headless mode that functions the same way as normal Chrome. ( https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c36 )

--headless=chrome

In your script, replace --headless with --headless=chrome to get the improved headless mode, which functions the same as regular Chrome:

chrome_options.add_argument("--headless=chrome")

Then your code will work as "perfectly" as it did in regular Chrome.