如何忽略已经访问过的域？ Java | Jsoup

Question

OK so I am starting a Bing search, then retrieving a couple resulting urls and using those as starting points to traverse other pages, parsing links from them and adding them to a List.

The problem I'm having is, I don't want to visit the same domain twice. I can stop it from visiting the same URL but if a page has link to another part of the website (such as an about page) I can't. Currently I've a LinkedList where I add a URL to every time I parse one from the document using Jsoup. And I have a HashMap for storing already visited URLs. So I have it set up in a basic "if" like this:

if(!urlsVisited.containsKey(url))
{
    urlsToVisit.add(url);
    urlsVisited.put(url, url); 
}

This is in a for loop where I retrieve the links on each page (currently 4 threads handling 4 pages).

This stops it from adding the likes of " http://www.stackoverflow.com " twice but doesn't work if I were to come across " http://www.stackoverflow.com/questions/ask ".

I would like to add one link from StackOverflow (for example) and then be done with that domain. Any ideas?

I'm using Jsoup api in Java to parse results.

Answer 1

Use the java.net.URL class to pull the host name, and use that as the key to your urlsVisited map.

http://docs.oracle.com/javase/6/docs/api/java/net/URL.html#getHost()

Answer 2

You can use URI class to parse your URLs. I also recommend to use Set<String> to store visited domains:

Set<String> urlsVisited = new HashSet<String>();
...

String domain = new URI(url).getHost();
if(!urlsVisited.contains(domain))
{
    urlsToVisit.add(url);
    urlsVisited.add(domain); 
}