在纯Java中检索呈现的HTML DOM

Question

I know there are already some similar questions here. But I do not want to build a browser in Java, I only want to see the source code fully generated (or "rendered"). As if I look at the generated DOM in the browser. Does anybody know a tool for that?

I had a look at Cobra and HtmlUnit , but they dont seem to be able to render more complex websites correctly. Especially if there are AJAX calls adding content to the site after it has loaded. I really need a tool that does the same as a browser does, without the actual display of it. Do I have to remote control a browser in the end?

Does anybody has experience with that?

A very similar question but without any satisfying answeres can be found here .

Answer 1

I don't believe that a library exists that does scraping of the asynchronous calls after the page is loaded.

My recommendation is:

1. Get the HTML of a page using Cobra or a similar library.
2. Parse the source for AJAX requests. (for example, the ajax call will have a URL parameter and a "data" JSON string you can use for the request)
3. For each AJAX call, make another request to the URL parameter you captured.
4. Append the result from each AJAX call to the source of your HTML from the original page.

It's not a perfect solution and it will not help you in the scenarios that require the user to trigger an event. Also your code for capturing the URLs for the AJAX events will differ depending on what javascript library the website is using to make its async calls.

Hope that helps.

Answer 2

I have to answer this myself... In the end the best solution I found was actually HtmlUnit. It is just too slow for my needs. So I built my own tool, that of course needs manual setup to call the required links. But thus it does not have to wait for any js timeouts or alike, but parses the requested information from the page and does the desired calls. Its a lot of manual work, but it looks like there is no other solution that works fast enough.

Answer 3

Selenium do some what similar to this. You need to install selenium remote control on your machine. Then you can pass url request to the selenium. Selenium will open a browser a render the html/dhtml page metioned in the url. After that you can get the entire dom by querying to the selenium. you can do all these thing by coding

http://seleniumhq.org/ please note: You need to install either slenium webdriver or selenium remote control, not selenium ide.