Python Selenium Firefox script crashing after restarting driver instance

Question

I have a Python Selenium script that is running in a loop, it quits the browser every 100 iterations like this...

def init_driver():
    ffprofile = webdriver.FirefoxProfile("my_profile");
    ffprofile.add_extension(extension="myaddon.xpi")
    return driver

driver = init_driver()

for i, item, in enumerate(item_list):
    check_item(item)
    print ( "" )
    if i == 0:
        print ( "Ignoring First Item" )
    elif i % 100 == 0:
            driver.quit()
            driver = init_driver()

During the restart of the driver it randomly crashes with an error...

Traceback (most recent call last):

  File "C:\scripts\main.py", line 118, in <module>
    driver = init_driver()

    File "C:\scripts\main.py", line 98, in init_driver
    driver = webdriver.Firefox(firefox_profile=ffprofile)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 78, in __init__
    self.binary, timeout)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\firefox\extension_connection.py", line 49, in __init__
    self.profile.add_extension()

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\firefox\firefox_profile.py", line 91, in add_extension
    self._install_extension(extension)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\firefox\firefox_profile.py", line 287, in _install_extension
    shutil.rmtree(tmpdir)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())

    File "C:\Users\john\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)

    PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\john\\AppData\\Local\\Temp\\tmpoaizz71l.webdriver.xpi\\platform\\WINNT_x86-msvc\\components\\imehandler.dll'

Sometimes it can go through thousands of iterations without crashing and other times it happens after 100

Anyone any ideas what is happening?

Answer 1

Likely, this happens on temporary profile removal, due to a race condition - a shared libtary still accesses the directory. Firefox and thus Selenium are quite asynchronous, so sometimes shit just happens on various stages of the process, you may look through the questions of the kind 'waiting for an element to be loaded' and alike.

I guess that you have two practical options to handle the issue: if this happens /rarely enough/, simply add a try-except-wait-repeat block to the code near driver_init . This is ugly, but sometimes it's the only reasonable way in terms of outcome-for-efforts.

Alternatively, you could switch the plaform :) In systems like Linux opened files usually don't block their removal.

Answer 2

Another workaround, along with @user3159253's answer , add a short wait period between driver.quit() and driver = init_driver() . If it's not time critical to open the new browser, maybe wait 0.5 or 1.0 seconds and increase/decrease until the error occurs.

If that's too much experimentation, combine this with the try...except block as suggested in his/her answer and start a timer before driver.quit() and check how much time has passed when the exception occurs.