Headless chrome cannot detect elements(selenium)
Question
It works well when done in non-headless mode but in headless mode no elements are detected.
code
url: https://moneyforward.com/users/sign_in
I wanna type into mail_address and password and click button in headless mode.
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import chromedriver_binary
options = Options()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(options=options)
driver.get("https://moneyforward.com/users/sign_in")
time.sleep(5)
# type into mail_address
driver.find_element_by_id('sign_in_session_service_email').send_keys({mail_address}])
# type into passowrd
driver.find_element_by_id('sign_in_session_service_password').send_keys({password})
# click login button
driver.find_element_by_id("login-btn-sumit").click()
The error statement is as follows.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="sign_in_session_service_email"]"}
(Session info: headless chrome=80.0.3987.132)
Why doesn't it work in headless mode when it works in non-headless mode? Help me.
3 answers
solution1
0 2020-03-10 18:24:40
Try to wait for the element rather than just using a hard coded sleep:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "sign_in_session_service_email"))
Before you try that you could crank up the sleep to a higher wait, jsut to debug.
It's also possible that in headless mode your "screen size" isn't configured correctly and is causing a responsive reaction to the screen causing the elements to change.
solution2
0 2020-03-10 18:35:45
It seems the page you are accessing does not allow headless browser mode. Try printing the HTML page and see if it returns all elements. I tried by adding the following statement and looks like the application returns the forbidden page.
print(driver.find_element_by_xpath('//*').get_attribute("innerHTML"))
I got this from the URL you have shared.
Forbidden
You can refer to Chrome headless options for debugging
solution3
0 ACCPTED 2020-03-10 21:07:16
I took your code and modified a bit and executed at my end and here is the execution result:
Code Block:
from selenium import webdriver options = webdriver.ChromeOptions() options.headless = True options.add_argument('window-size=1400,600') options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) driver = webdriver.Chrome(options=options, executable_path=r'C:\\WebDrivers\\chromedriver.exe') driver.get('https://moneyforward.com/users/sign_in') print(driver.page_source) driver.save_screenshot('./save_screenshot_method.png') #Capture the screen driver.quit()Console Output:
<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">Forbidden</pre></body></html>Browser Snapshot:
Analysis
It seems ChromeDriver driven google-chrome-headless is getting detected and being Access Denied the message Forbidden is shown.
Solution
As a solution you can adopt some strategies so ChromeDriver driven Chrome Browsing context doesn't gets detected and you can find a couple of detailed discussions in:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.