Error while using Firefox headless, Selenium and Python

Question

I am trying to use firefox headless, Selenium framework and Python to fetch webpage on Amazon EC2 Ubuntu linux. My code looks like this:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True

driver = webdriver.Firefox(options=options,executable_path='/home/ubuntu/geckodriver')
driver.get("https://google.com")
print('Done')
driver.quit()

Now when I run this, I get the following error:

Traceback (most recent call last):
  File "test1.py", line 7, in <module>
    driver = webdriver.Firefox(options=options,executable_path='/home/ubuntu/geckodriver')
  File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 170, in __init__
    RemoteWebDriver.__init__(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Connection refused (os error 111)

I have made sure that my geckodriver and firefox versions are compatible, I have tried rebooting my EC2 instance but nothing is working.

Any help is appreciated.

Answer 1

try this [with webdriver-manager ]

pip install webdriver-manager 
from webdriver_manager.firefox import GeckoDriverManager
self.browser = webdriver.Firefox(executable_path=GeckoDriverManager().install())

and it will automatically fix any driver error you have

Answer 2

Could that be used.

from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument('--headless')
driver = webdriver.Firefox(executable_path='path to the driver', options=options)

Answer 3

This is the complete working code, I tested it on Windows machine with Pycharm community edition IDE

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from webdriver_manager.firefox import GeckoDriverManager
options = Options()
options.headless = True
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install(),firefox_options=options)
driver.get("https://google.com")
print('Done')
driver.quit()

Answer 4

Update: It seems like it was an OS issue. When I created a new EC2 instance using Amazon Linux, same code is working without issue. The older EC2 instance (Ubuntu) is still giving me the same error.

Answer 5

You need to pass the complete url including the www . So instead of:

driver.get("https://google.com")

You need:

driver.get("https://www.google.com/")