使用Nutch检索页面内容

Question

I have a very large list of seeds to be crawled (only those seeds are needed without any deepening). How can I use Nutch to retrieve:

1. the HTML of
2. the text content of
3. (Preferably) the out-links of

the seed pages? (without any indexing and integration into any other platform like Solr).

Thanks

Answer 1

Well, there are many issues you want to address. Below are the issues with their solutions:

1. Limiting crawling to seed list : enable the scoring-depth plugin and configure it to allow only 1 level of crawling.
2. Getting textual content : Nutch does that by default.
3. Getting HTML raw data : it is not possible by Nutch 1.9. You need to download Nutch from its trunk repository and build it because the HTML content is scheduled for Nutch's next release (1.10).
4. Extracting outlinks : you can do that, but you have to write a new indexingFilter to index the outlinks.
5. Doing all of the above without Solr : you can do that. However, you have to write a new indexer that stores the extract data in whatever format you want.