Java Selenium Chromedriver webdriver尽快

Question

I am connecting a standalone program to a website, and I have to read some pages of the website. For first, I used Jsoup, but with this I discovered that some informations that I need are loaded after page load, so I searched for webdrivers. (I am not looking for images or something big, my content are all textual)

Now, i found the ChromeDriver, but it is too slow for my case because it has a lot of options and features. In my case I need just a step more than the Jsoup possibilities. It is possible to disable the best part of ChromeDriver options and features to reach this goal?

For example, i saw plugin disabling, but is one by one and is not for every chrome browser on every pc. I didn't found an option like "plugin.disable-all".

Furthermore,in this way I cannot open more than a few instance of chromedriver. In this moment, every instance of the chromedriver is opening a Google Chrome Helper that uses 100Mb of ram.

Hope all is clear

Answer 1

HtmlUnit might be enough for your needs. It does support some Javascript.

It can be used with Webdriver . But might as well be enough on its own

Answer 2

To make your webdriver run faster (but not that much faster), you can run the driver in Headless mode. See these articles for a tutorial on how to go into Headless mode for Chrome.

Before starting the driver, add the --headless argument to ChromeOptions .

Headless mode can speed up your automation by not rendering the browser window, but keep in mind that doing a straight HTTP GET with JSoup would always be faster.

My advice would be to reverse engineer the page a bit more, and see if you can figure out how to query directly whatever the (presumably AJAX) calls are putting on the page. If you can treat those specific requests as an API and only query for exactly what you want, you will be able to get results faster than with browser automation through Selenium.