Download HTML of a webpage that is changed by JavaScript

Question

The program I am writing is in Java.

I am writing a little program that will download the html of webpages and save them. It works easily for basic pages that don't use JavaScript. But how can I download the page if I want it after a script has updated it? The page I am dealing with is actually updated by Ajax which might be one step harder.

I understand that this is probably a difficult problem that involves setting up a JavaScript run time environment of some kind. I am prepared for a solution of any level of difficulty, I just don't know exactly how to approach it or where to get started.

Answer 1

You can't do that alone with Java only. As the page that you want to download is rendered with javascript, then you must be able to execute the javascript to get the whole rendered page.

Because of this situation, you need to use a headless browser which is a web browser that can access to web pages but can't show the output within a GUI, aims to provide the content of web pages as fully rendered to serve to the programs or scripts.

You can start with the most famous ones which are Selenium , HtmlUnit and PhantomJS