Selenium java silent mode

Question

I need to use Selenium to parse few thousand pages per day.

Everything works fine, but every parse of page need to reopen browser, and it take time.

Do you know how to enable Selenium in silent mode in java to speed up parsing ?

Thanks

Answer 1

If you're worried about the speed of opening a thousand browsers, you could switch over to using the HtmlUnitDriver class as your WebDriver , since it's headless and takes far less time to spin up and shut down than a full browser instance.

EDIT : I'm assuming you need the browser to close in order to reset the state of the session? If so, then you could feasibly overcome this requirement by structuring your code to cleanly destroy the session, instead of closing the browser entirely. If your session is maintained by a cookie, then you can do:

driver.manage().deleteAllCookies(); // or
driver.manage().deleteCookieNamed("JSESSIONID"); // if using J2EE, for example

If you don't need the browser to close, then why on earth aren't you reusing the same WebDriver ?

Answer 2

Don't open a new browser for each page? Just reuse the current browser instance for all pages.

FirefoxDriver driver = new FirefoxDriver();
List<String> urls = new ArrayList<>();
// load the urls List
for (String url : urls)
{
    driver.get(url);
    // do stuff to page
}

I made up some List<> to store the arrays. You would obviously replace that with whatever data type you have storing the URLs that you need to deal with and loop through those.