Why ChromeDriver crashed? (Python3 Selenuim Ubuntu)

Question

I want to Web Scraping. I use ubuntu 20.04, python 3.8, selenium, ChromeDriver. But why ChromeDriver crashed? Is there something wrong with the version?

I've tried various Chrome options, --no-sandbox, --disable-gpu, --disable-dev-shm-usage but it hasn't changed.

• code

# - coding: utf-8 --
import time
import random
import requests
import math
import re
import time
import datetime
from bs4 import BeautifulSoup
from sqlalchemy import create_engine
from string import Template
from selenium import webdriver
from selenium.webdriver import ChromeOptions
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

def hoge(url):
    options = ChromeOptions()
    options.add_argument('--headless')
    serv = Service(ChromeDriverManager().install())
    driver = webdriver.Chrome(service=serv, options=options)
    driver.get(url)

• Environment

Ubuntu 20.04
Python 3.8.10
pip 21.2.3 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
selenium 4.1.0
Google Chrome 97.0.4692.71
ChromeDriver 97.0.4692.71

• ErrorMessage

====== WebDriver manager ======
Current google-chrome version is 97.0.4692
Get LATEST chromedriver version for 97.0.4692 google-chrome
Driver [/root/.wdm/drivers/chromedriver/linux64/97.0.4692.71/chromedriver] found in cache
Traceback (most recent call last):
  File "/var/www/hoge.py", line 271, in <module>
    hoge("https://google.com")
  File "/var/www/hoge.py", line 190, in hoge
    driver = webdriver.Chrome(service=serv, options=options)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
    super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/chromium/webdriver.py", line 93, in __init__
    RemoteWebDriver.__init__(
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 268, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 359, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x55f08dc53a23 <unknown>
#1 0x55f08d71ee18 <unknown>
#2 0x55f08d7421f1 <unknown>
#3 0x55f08d73d91a <unknown>
#4 0x55f08d77874a <unknown>
#5 0x55f08d772883 <unknown>
#6 0x55f08d7483fa <unknown>
#7 0x55f08d7494c5 <unknown>
#8 0x55f08dc8316d <unknown>
#9 0x55f08dc995bb <unknown>
#10 0x55f08dc84e75 <unknown>
#11 0x55f08dc99e85 <unknown>
#12 0x55f08dc7886f <unknown>
#13 0x55f08dcb4ae8 <unknown>
#14 0x55f08dcb4c68 <unknown>
#15 0x55f08dccfaad <unknown>
#16 0x7efe8ed65609 <unknown>

• pip list

Package             Version
------------------- --------------
async-generator     1.10
attrs               21.3.0
beautifulsoup4      4.10.0
certifi             2021.10.8
cffi                1.15.0
charset-normalizer  2.0.9
chromedriver-binary 96.0.4664.45.0
colorama            0.4.4
configparser        5.2.0
crayons             0.4.0
cryptography        36.0.1
greenlet            1.1.2
h11                 0.12.0
idna                3.3
lxml                4.7.1
outcome             1.1.0
pip                 21.2.3
psycopg2-binary     2.9.2
pycparser           2.21
pyOpenSSL           21.0.0
requests            2.26.0
selenium            4.1.0
setuptools          57.4.0
six                 1.16.0
sniffio             1.2.0
sortedcontainers    2.4.0
soupsieve           2.3.1
SQLAlchemy          1.4.29
trio                0.19.0
trio-websocket      0.9.2
urllib3             1.26.7
webdriver-manager   3.5.2
wsproto             1.0.0

• pip freeze

async-generator==1.10
attrs==21.3.0
beautifulsoup4==4.10.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.9
chromedriver-binary==96.0.4664.45.0
colorama==0.4.4
configparser==5.2.0
crayons==0.4.0
cryptography==36.0.1
greenlet==1.1.2
h11==0.12.0
idna==3.3
lxml==4.7.1
outcome==1.1.0
psycopg2-binary==2.9.2
pycparser==2.21
pyOpenSSL==21.0.0
requests==2.26.0
selenium==4.1.0
six==1.16.0
sniffio==1.2.0
sortedcontainers==2.4.0
soupsieve==2.3.1
SQLAlchemy==1.4.29
trio==0.19.0
trio-websocket==0.9.2
urllib3==1.26.7
webdriver-manager==3.5.2
wsproto==1.0.0

What else should I try?

Answer 1

Perhaps you should try with something simpler like:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://google.com")

btn = driver.find_element(By.XPATH, '//*[@id="L2AGLb"]')
btn.click()

This is a simple program that starts chrome with selenium, goes to google.com and clicks the button to accept the cookies. It worked for me with the same version of the ChromeDriver you have.