Selenium running in headless mode throwing error

Question

I am running a flask application with selenium in headless mode. Sometimes its throwing the following error. Not always.

raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
  (Session info: headless chrome=91.0.4472.101)

Code

from flask import Flask, jsonify, request, render_template, redirect, url_for
from selenium import webdriver
import validators

options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
driver = webdriver.Chrome(executable_path = './static/chromedriver', options = options)
@app.route('/',methods = ['GET','POST'])
def parseurl():
     if request.method == 'GET':
          print("GET REq processing")
          return render_template('index.html')
     else:
          if "url" in request.form and "name" in request.form:
               URL = request.form.get("url")
               if validators.url(URL):
                    org_name = request.form.get("name")
                    driver.get(URL)
                    driver.implicitly_wait(2)
                    data = driver.page_source
     return render_template('index.html',result = data)

This error comes in replit server with the following two urls

https://www.tucsonsymphony.org/event/mozart-and-prokofiev/2022-03-06/ https://mobilesymphony.org/event/a-john-williams-jubilee

These two links are working fine in local machine.

Other urls like https://alabamasymphony.org/event/beginning are working fine. chrome driver used is ChromeDriver 91.0.4472.101

Answer 1

It's definitely not the cleanest solution but as it's only occurring occasionally you could add in an exception handler, which would prevent the driver from crashing:

from flask import Flask, jsonify, request, render_template, redirect, url_for
from selenium import webdriver
import validators
from selenium.common.exceptions import WebDriverException

options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
driver = webdriver.Chrome(executable_path = './static/chromedriver', options = options)
@app.route('/',methods = ['GET','POST'])
def parseurl():
    if request.method == 'GET':
        print("GET REq processing")
        return render_template('index.html')
    else:
        if "url" in request.form and "name" in request.form:
            URL = request.form.get("url")
            if validators.url(URL):
                try:
                    org_name = request.form.get("name")
                    driver.get(URL)
                    driver.implicitly_wait(2)
                    data = driver.page_source
                except WebDriverException:
                    # reset the webdriver
                    driver.quit()
                    driver = webdriver.Chrome(executable_path = './static/chromedriver', options = options)
     return render_template('index.html',result = data