如何从不同的文件读取URL并设置不同的深度进行爬网？

Question

I want to have two files seed.txt and seed2.txt and in each file to have different urls. In seed.txt the depth for crawling i want to be for ex. 2 and in seed2.txt the depth to be 3.
Is there any solution or workaround to do this??

Answer 1

I want to have two files seed.txt and seed2.txt and in each file to have different urls

You need to maintain the seed file name as is; do not rename it to seed2 etc. Instead, You can create two seperate urls directory with a seed file in each containing different set of urls. Ex. folder 'urls1' will have one seed.txt and another folder 'urls2' will have another seed.txt with a different set of urls. But also make sure to create seperate crawl directories where the crawl data would go to (ex. create a 'crawl1' directory for seed.txt in 'urls1' folder and 'crawl2' directory for the 'seeds.txt' in 'urls2'.

In seed.txt the depth for crawling i want to be for ex. 2 and in seed2.txt the depth to be 3.

You should specify the depth value in your crawl command not in the seed.txt. In your case, run the following commands in seperate terminals if running on the same machine (provided your nutch/hadoop configuration supports running multiple crawl jobs in parallel.

• bin/nutch crawl urls1 -dir crawl1 -depth 2

• bin/nutch crawl urls2 -dir crawl2 -depth 3

Hope this helped!