如何使用jsoup获取到站点页面的可导航链接？

Question

I am implementing a basic crawler with the purpose of later use in a vulnerability scanner. I am using jsoup for the connection/retrieving and parsing of html document.

I supply manually the base/root of the intended site(www.example.com) and connect.

...
Connection connection = Jsoup.connect(url).userAgent(USER_AGENT);
Document htmlDocument = connection.get();
this.htmlDocument = htmlDocument;
...

Then i retrieve all the links on the page.

...
Elements linksOnPage = htmlDocument.select("a[href]");
... 

After this I loop between the links and try to get the links to all the pages on the site.

for (Element link : linksOnPage) {
                this.links.add(link.absUrl("href"));
    }

The problem is as follows. Depending on the links I get, some might not be links to new pages or not even links to pages at all. As an example a got links like:

• https://example.example.com/webmail

• http://193.231.21.13

• mailto:example.example@exampl.com

What i need some help whit is the filtering of the links so that i get only links to new pages of the same root/base site.

Answer 1

This is easy. Check if absUrl ends with image format or js or css:

if(absUrl.startsWith("http://www.ics.uci.edu/") && !absUrl.matches(".*\\.(bmp|gif|jpg|png|js|css)$")) 
{

    //here absUrl starts with domain name and is not image or js or css
}