无法通过 Python 中的 Selenium 杀死 Chrome 进程并使用 ChromeDriver 和 Chrome 耗尽内存

Question

I have a crawling process that kicks off selenium in a custom class that looks like this:

class BrowserInterface:

def __init__(self, base_url, proxy_settings):

    self.base_url = base_url

    self.display = Display(visible=0, size=(1024, 768))
    self.display.start()

    proxy_argument = '--proxy-server={0}'.format(PROXY_URL.format(
        proxy_settings.get('proxy_host'),
        proxy_settings.get('proxy_port')
    ))

    logger.debug(proxy_argument)

    options = webdriver.ChromeOptions()
    options.add_argument('--no-sandbox')
    options.add_argument(proxy_argument)

    selenium_chrome_driver_path = os.path.join(settings.DEFAULT_DRIVER_PATH,
                                               settings.CHROME_DRIVERS[settings.CURRENT_OS])

    self.driver = webdriver.Chrome(executable_path=selenium_chrome_driver_path, chrome_options=options)

def visit(self, url):
    url = urljoin(self.base_url, url)
    self.driver.get(url)

def body(self):
    soup = BeautifulSoup(self.driver.page_source)
    return soup.find("body").text

def quit(self):
    self.driver.quit()
    self.display.stop()

This BrowserInterface class is initialized in a batch queue and the quit() method is called at the end of the batch. There are no issues starting chrome and getting the data. The trouble is, at the end of each job when the quit() method is called chrome goes into zombie mode. When the next BrowserInterface is initialized it starts a new chrome instance. Due to this, the box is running out of memory. I've tried running the a kill command as well on the chrome process but it stays running. Any direction would be greatly appreciated as I'm about to pull my hair out over this.

Running on Ubuntu 18.04, Google Chrome 70.0.3538.110, ChromeDriver 2.44, Python3.6.6

Thanks in advance!

Answer 1

From your code trials it is pretty much evident you have invoked self.driver.quit() which should have worked perfect.

However as the box is still running out of memory due to zombie chrome processes you took the right approach to execute the a kill command and you can add the following solution within the quit() method:

from selenium import webdriver
import psutil

driver = webdriver.Chrome()
driver.get('http://google.com/')

PROCNAME = "chrome" # to clean up zombie Chrome browser
#PROCNAME = "chromedriver" # to clean up zombie ChromeDriver
for proc in psutil.process_iter():
    # check whether the process name matches
    if proc.name() == PROCNAME:
        proc.kill()

Answer 2

Please see: https://stackoverflow.com/a/49756925

Creating bash as the root process allows for better 'zombie handling'. Python isn't meant to be run as a top level process which causes the zombies.