如何设置简单的JAVA Web爬虫的深度

Question

I wrote a simple recursive web crawler to fetch just the URL links from the web page recursively.

Now I am trying to figure out a way to limit the crawler using depth but I am not sure how to limit the crawler by specific depth (I can limit the crawler by top N links but I want to limit using depth)

For Ex: Depth 2 should fetch Parent links -> children(s) links--> children(s) link

Any inputs is appreciated.

    public class SimpleCrawler {

    static Map<String, String> retMap = new ConcurrentHashMap<String, String>();    

        public static void main(String args[]) throws IOException {
         StringBuffer sb = new StringBuffer();  
         Map<String, String> map = (returnURL("http://www.google.com"));
         recursiveCrawl(map);
          for (Map.Entry<String, String> entry : retMap.entrySet()) {
            sb.append(entry.getKey());
          }
        }

        public static void recursiveCrawl(Map<String, String> map)
                throws IOException {
            for (Map.Entry<String, String> entry : map.entrySet()) {
                String key = entry.getKey();
                Map<String, String> recurSive = returnURL(key);
                recursiveCrawl(recurSive);
            }
        }

        public synchronized static Map<String, String> returnURL(String URL)
                throws IOException {

            Map<String, String> tempMap = new HashMap<String, String>();
            Document doc = null;
            if (URL != null && !URL.equals("") && !retMap.containsKey(URL)) {
                System.out.println("Processing==>" + URL);
                try {
                    URL url = new URL(URL);
                    System.setProperty("http.proxyHost", "proxy");
                    System.setProperty("http.proxyPort", "port");
                    doc = Jsoup.connect(URL).get();
                    if (doc != null) {
                        Elements links = doc.select("a");
                        String FinalString = "";
                        for (Element e : links) {
                            FinalString = "http:" + e.attr("href");
                            if (!retMap.containsKey(FinalString)) {
                                tempMap.put(FinalString, FinalString);
                            }
                        }
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
                retMap.put(URL, URL);
            } else {
                System.out.println("****Skipping URL****" + URL);
            }
            return tempMap;
        }

    }

EDIT1:

I thought of using worklist hence modified the code. I am not exactly sure how to set depth here too (I can set the number of webpages to crawl but not exactly depth). Any suggestions would be appreciated.

public void startCrawl(String url) {
        while (this.pagesVisited.size() < 2) {
            String currentUrl;
            SpiderLeg leg = new SpiderLeg();
            if (this.pagesToVisit.isEmpty()) {
                currentUrl = url;
                this.pagesVisited.add(url);
            } else {
                currentUrl = this.nextUrl();
            }
            leg.crawl(currentUrl);
            System.out.println("pagesToVisit Size" + pagesToVisit.size());
            // SpiderLeg
            this.pagesToVisit.addAll(leg.getLinks());
        }
        System.out.println("\n**Done** Visited " + this.pagesVisited.size()
                + " web page(s)");
    }

Answer 1

Based on the non-recursive approach:

Keep a worklist of URLs pagesToCrawl of type CrawlURL

class CrawlURL {
  public String url;
  public int depth;

  public CrawlURL(String url, int depth) {
    this.url = url;
    this.depth = depth;
  }
}

initially (before entering the loop):

Queue<CrawlURL> pagesToCrawl = new LinkedList<>();
pagesToCrawl.add(new CrawlURL(rootUrl, 0)); //rootUrl is the url to start from

now the loop:

while (!pagesToCrawl.isEmpty()) { // will proceed at least once (for rootUrl)
  CrawlURL currentUrl = pagesToCrawl.remove();
  //analyze the url
  //updated with crawled links
}

and the updating with links:

if (currentUrl.depth < 2) {
  for (String url : leg.getLinks()) { // referring to your analysis result
    pagesToCrawl.add(new CrawlURL(url, currentUrl.depth + 1));
  }
}

You could enhance CrawlURL with other meta data (eg link name, referrer,. etc.).

Alternative: In my upper comment I mentioned a generational approach. Its a bit more complex than this one. The basic Idea is to keep to lists ( currentPagesToCrawl and futurePagesToCrawl ) together with a generation variable (starting with 0 and increasing every time currentPagesToCrawl gets empty). All crawled urls are put into the futurePagesToCrawl queue and if currentPagesToCrawl both lists will be switched. This is done until the generation variable reaches 2.

Answer 2

You could add a depth parameter on the signature of your recursive method eg

on your main

recursiveCrawl(map,0);

and

public static void recursiveCrawl(Map<String, String> map, int depth) throws IOException {
    if (depth++ < DESIRED_DEPTH) //assuming initial depth = 0
        for (Map.Entry<String, String> entry : map.entrySet()) {
            String key = entry.getKey();
            Map<String, String> recurSive = returnURL(key);
            recursiveCrawl(recurSive, depth);
        }
    }
]

Answer 3

You can do something like this:

static int maxLevels = 10;

public static void main(String args[]) throws IOException {
     ...
     recursiveCrawl(map,0);
     ...
}

public static void recursiveCrawl(Map<String, String> map, int level) throws IOException {
    for (Map.Entry<String, String> entry : map.entrySet()) {
        String key = entry.getKey();
        Map<String, String> recurSive = returnURL(key);
        if (level < maxLevels) {
            recursiveCrawl(recurSive, ++level);
        }
    }
}

Also, you can use a Set instead of a Map .