在检索其内容之前执行页面脚本

Question

I've a page where i need to automate some tasks and scrape some data, but the page runs some JS after loading to inject some data into the DOM; that i cannot intercept (not in a good format anyway), I was hoping to find a solution that is fast and not memory consuming.

I've attempt to get the scripts myself and execute them using some headless driver (namely phantomJs) but it didn't update the page source and i'm not sure how to retrive the updated DOM from that

var page = GetWebPage(url);
var scripts = page.Html.QuerySelectorAll("script");

var phantomDriver = new PhantomJSDriver(PhantomJSDriverService.CreateDefaultService(Directory.GetCurrentDirectory()));
phantomDriver.Navigate().GoToUrl(url);

foreach (var script in scripts)
    phantomDriver.ExecuteScript(script.InnerText);

var at = phantomDriver.PageSource;

Answer 1

You could use a 'wait'. According to this link , Selenium has both implicit and explicit waits. The example below is using an explicit wait.

To use an explicit wait, use WebDriverWait and ExpectedConditions . I'm not sure what language you're using but here's an example in python. This uses WebDriverWait in a try-catch block, allowing timeout seconds to meet the specified ExpectedConditions . As at June 2019, conditions are available in:

• Java ;
• Python ; and
• .NET

Example code in python:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

url = 'https://stackoverflow.com/questions/56724178/executing-page-scripts-before-retrieving-its-contents'
target = (By.XPATH, "//div[@class='gravatar-wrapper-32']")
timeout = 20  # Allow max 20 seconds to find the target

browser = webdriver.Chrome()
browser.get(url)
try:
    WebDriverWait(browser, timeout).until(EC.visibility_of_element_located(target))
except TimeoutException:
    print("Timed out waiting for page to load")
    browser.quit()

The important bit is between try and except which you would modify to use the specific 'expected condition' you're interested in.