How to allow apache nutch to crawl forever

Question

I am using apache nutch (2.2.1) for crawling. What changes are need if I want to crawl forever. Guide me completely as I am not familiar with nutch too mutch.

Answer 1

If you want to crawl forever, the following is the script you need:

#!/bin/bash

./bin/nutch inject urls #urls is the seed data
while [ 1 == 1 ]
do
   ./bin/nutch generate -topN 10000 # 10000 is the number of URLs will be fetch in each crawling round, you can modify it
   ./bin/nutch fetch -all
   ./bin/nutch parse -all
   ./bin/nutch updatedb

done

Hope this helps

Le Quoc Do