如何使用crawler4j解析文档

Question

I wanted to parse all the documents containing some text I enter as "query" using crawler4j in Eclipse.

Any ideas?

Answer 1

Not really a "direct" answer, but I also played with crawling these last few days. I looked first at Crawler4J, then stumbled on JSoup . Did not play much with the crawler, but jSoup turns out to be quite an easy tool for parsing. Hence my suggestion. I guess crawler is good if you really need to crawl a part of the web. But JSoup really seems to shine as a good parser. Similar to JQuery in terms of selecting nodes etc... So perhaps use the crawler for first collecting documents, then parse them using JSoup. Here's a quick example:

    Document doc = Jsoup.connect("http://example.com").userAgent("Mozilla").timeout(5000)
            .get();
    Elements els = doc.select("li");