scrape website multiple pages using Web Client java

Question

I am trying to scrape a website, using Web Client, i am able to get the data on the first page and parse it, but I do not know how to read the data on the second page, the website is calling a java script to navigate to the second page. Can anyone suggest me how do I get the data from the next pages?

Thanks in advance

Answer 1

The problem you're going to have is while you (a person) can read the JavaScript in the first page and see it is navigating to another page, having the computer do this is going to be hard.

If you could identify the block of code performing the navigation, you would then need to execute it in such a way that allowed your program to extract the URL. This again is going to be very specific to the structure of the JavaScript and would require a person to identify this.

In short, I think you're dead in the water with this one, though it serves as a good example of why the Unobtrusive JavaScript concept is so important.

Answer 2

该框架将HtmlUnit与其无头的启用JavaScript的浏览器集成在一起，以完全支持在同一WebClient会话中编写多个页面： https : //github.com/subes/invesdwin-webproxy