如何减少 selenium python 中的 chromedriver cpu 使用率？

Question

i'm trying to automate login and scraping data using selenium in python im using chromedriver as a driver. but i want to do that with 5 account, im still figuring out what is the best method to do.

for now, i create the code in python file then i create batch file to run the python. so i can open multiple batch file with multiple account.

but the problem is cpu usage is too high so im just able to do with 3 account.

Troubleshooting i do so far is changing options to make chrome headless using this code

    self.options = webdriver.ChromeOptions()
    self.options.headless = True
    self.options.add_argument(f'user-agent={user_agent}')
    self.options.add_argument("--window-size=1920,1080")
    self.options.add_argument('--ignore-certificate-errors')
    self.options.add_argument('--allow-running-insecure-content')
    self.options.add_argument("--disable-extensions")
    self.options.add_argument("--proxy-server='direct://'")
    self.options.add_argument("--proxy-bypass-list=*")
    self.options.add_argument("--start-maximized")
    self.options.add_argument('--disable-gpu')
    self.options.add_argument('--disable-dev-shm-usage')
    self.options.add_argument("--FontRenderHinting[none]")
    self.options.add_argument('--no-sandbox')
    self.options.add_argument('log-level=3')
    self.driver = webdriver.Chrome(executable_path="chromedriver.exe", options=self.options)

but still cant achieve my target(5 account). is there anything i can do to achieve my target?

Thanks.

Answer 1

I have noticed (by watching htop processes while tests are running), the crash reporting of chrome takes up a huge amount of CPU and memory

Since you are using Chrome Headless, I've found adding this reduces the CPU usage by about 20% for me:

--disable-crash-reporter

This will only disable when you are running in headless This might speed things up for you!!!

My settings are currently as follows and I reduce the CPU (but only a marginal time saving) by about 20%:

    self.options.add_argument("--no-sandbox");
    self.options.add_argument("--disable-dev-shm-usage");
    self.options.add_argument("--disable-renderer-backgrounding");
    self.options.add_argument("--disable-background-timer-throttling");
    self.options.add_argument("--disable-backgrounding-occluded-windows");
    self.options.add_argument("--disable-client-side-phishing-detection");
    self.options.add_argument("--disable-crash-reporter");
    self.options.add_argument("--disable-oopr-debug-crash-dump");
    self.options.add_argument("--no-crash-upload");
    self.options.add_argument("--disable-gpu");
    self.options.add_argument("--disable-extensions");
    self.options.add_argument("--disable-low-res-tiling");
    self.options.add_argument("--log-level=3");
    self.options.add_argument("--silent");

I found this to be a pretty good list (full list I think) of command line switches with explanations: https://peter.sh/experiments/chromium-command-line-switches/

Some additional things you can turn off are also mentioned here: https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md

I hope this helps someone

Answer 2

self.options.add_argument('--headless')

this will make the driver run way faster then before