获取用于制作网络爬虫的页面的完整html源代码

Question

I'm trying to make a web crawler in java which takes URL of a webpage and navigates to other pages which are present in the source code of given web page. The issue is, i'm getting the source code of HTML with the help of jsoup which contains various tags like frames and some javascript file names. Now to navigate to other pages i need to access the http links given in frames and javascript files. How should i get those links in a list.

Answer 1

You need to do it recursively... found a frame tag/Element in a DOM Object, time to fetch DOM of its "src" attribute, keep doing it, store all the links that you find in subsequent fetching to an array.
You can use new threads to fetch frames DOM. Just to make whole process little bit faster.