Web頁面的動態內容未完全使用Htmlunit WebClient加載

Question

我正在嘗試使用HtmlUnit WebClient加載網頁（ https://genpact.taleo.net/careersection/sgy_external_career_section/jobsearch.ftl?lang=en ）進行抓取。 但內容未正確加載。 例如，我無法找到“應用”按鈕。 我的webclient代碼如下

webClient.setCssErrorHandler(new DefaultCssErrorHandler());
        webClient.setJavaScriptErrorListener(new DefaultJavaScriptErrorListener());
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());
        webClient.getCookieManager().setCookiesEnabled(true);
        webClient.waitForBackgroundJavaScript(60000);

有人可以幫我這個

Answer 1

這適合我

public static void main(String[] args) throws IOException{
    final String url = "https://genpact.taleo.net/careersection/sgy_external_career_section/jobsearch.ftl?lang=en";

    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_60)) {
        HtmlPage page = webClient.getPage(url);

        // waitForBackgroundJavaScript has to be called after every action
        // this page is really slow wait for the last part of the dynamic content
        while(!page.asText().contains("Previous\r\n1\r\n2\r\n3\r\n4\r\n")) {
            webClient.waitForBackgroundJavaScript(1_000);
        }

        System.out.println("-------------------------------------------------------------------------------");
        System.out.println(page.asText());
        System.out.println("-------------------------------------------------------------------------------");
    }
}