python selenium: how to find elements missing due to headless mode

Question

I am looking for a specific button on a webpage. See below a working minimal example:

from time import sleep

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from pathlib import Path


start_url = "https://flatfox.ch/en/search/?east=7.641746&north=47.589902&query=Basel&south=47.519342&west=7.547066"

options = Options()
options.headless = False

driverpath = Path(r'enter gecko driver path here') / 'geckodriver.exe'
driver = webdriver.Firefox(executable_path=driverpath,
                           options=options)

driver.get(start_url)
sleep(3)

buttons = driver.find_elements_by_css_selector('.button.button--primary')
print(len(buttons)) # we expect two buttons
>>2

however if i switch to headless, the button is not found anymore:

from time import sleep

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from pathlib import Path


start_url = "https://flatfox.ch/en/search/?east=7.641746&north=47.589902&query=Basel&south=47.519342&west=7.547066"

options = Options()
options.headless = True

driverpath = Path(r'enter gecko driver path here') / 'geckodriver.exe'
driver = webdriver.Firefox(executable_path=driverpath,
                           options=options)

driver.get(start_url)
sleep(3)

buttons = driver.find_elements_by_css_selector('.button.button--primary')
print(len(buttons)) # we expect two buttons
>>0

what can I do to find that button in headless mode?

so far tried:

options.add_argument('--window-size=1920,1080')
profile.set_preference("dom.webdriver.enabled", False)
profile.set_preference('useAutomationExtension', False)

Thanks for help!

Answer 1

affirmative, same issue, but with chrome driver all good? does it make difference for you to use Chrome or Firefox: I would suggest using python library webdriver_manager:

pip install webdriver-manager

it allows you install driver bin dynamically by commands:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

or

from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager

driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())

and use options like this:

options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options)

Answer 2

options = webdriver.ChromeOptions()
#options.headless = True
options.add_argument("--window-size=1920,1080")
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
browser = webdriver.Chrome(options=options)

Change the useragent as some browsers use it to detect headless mode and behaves differently

Headless browser detection