Nutch how to crawl all links from one website?
Question
for now i use the follow commands to crawl a website:
bin/nutch generate -topN 20
bin/nutch fetch -all
bin/nutch parse -all
bin/nutch updatedb
but with this method it takes ages before i have all links from that website. I want to crawl one website and get all the links.
how can i achieve this?
1 answers
solution1
0 2014-04-04 11:07:55
bin/nutch crawl是您要查找的命令
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Nutch does not crawl all links in form
How to crawl a website that has SAML authentication using ManifoldCF or nutch?
How to crawl images in Nutch?
Nutch didn't crawl all URLs from the seed.txt
Nutch - Crawl a page for links, but don't index
apache nutch don't crawl website
Single Crawl script to Crawl website (Nutch) and Index results (Solr)
How to crawl magnet links with Apache Nutch and Solr so that they're available in Solr query results?
How to Set topN via nutch crawl SCRIPT
How to config Nutch to crawl only the URLs in seeklist? (no crawl back need)
Related Tags