Headless Chrome - 获取空白页面源

Question

I'm trying to load a website with Chrome browser in headless mode using Selenium web driver. I face an issue with some specific websites. The page is loading, in the first 2-3 seconds it shows a page with "please enable javascript..." and after 3 seconds, page source goes blank.

I'm using Selenium and especially Chrome for long time and I am familiar with the platform. For the purpose of this case, I'm using Chrome Version 73.0.3683.86 , ChromeDriver 2.46.628411 (which is compatible according to Which ChromeDriver version is compatible with which Chrome Browser version? ) on a Mac OS. selenium java version is latest - 3.141.59

I suspect that headless Chrome cannot handle specific content-type such as "svg" and any other GUI related HTTP response.

    ChromeOptions chromeOptions = new ChromeOptions();
    chromeOptions.addArguments("--headless");
    WebDriver driver = new ChromeDriver(chromeOptions);

    driver.get("https://identity.tescobank.com/login");

    Thread.sleep(3000);

    System.out.println(driver.getPageSource());

    driver.quit();

Expected result is to have the page source same as it is showing in non-headless mode.

Answer 1

Headless Chrome should be able to handle everything the normal Chrome can do:

It brings all modern web platform features provided by Chromium and the Blink rendering engine to the command line.

(see https://developers.google.com/web/updates/2017/04/headless-chrome )

Since only the login page of a bank causes you trouble, my guess is that the security of the page detects an anomaly and decides not to serve you.

One way they can do that is by looking at the User Agent string which contains HeadlessChrome .

That said, unless you're writing integration tests for the bank, your behavior is at least suspicious. If you have a valid and legal concern, clear it with the bank first. They might take actions against you, otherwise. Blocking your IP address (which could affect many people) or asking the police to have a word with you.

Answer 2

I was facing similar issue in my script, after login. Somehow refreshing the page resolved the issue. driver.navigate().refresh();