Scrapy搜寻器-创建10,000个蜘蛛还是一个蜘蛛爬行10,000个域？

Question

I need to crawl upto 10,000 websites

since every website is unique with its own HTML structure and requires its own logic of XPATH & creating and delegating Request objects. I'm tempted to create a unique spider for each website

But is this the best way forward?. Should i perhaps have a single spider and add all the 10,000 websites in the start_urls and allowed_domains , write scraping libraries and go for it?

which is the best practice in regards to this?

Answer 1

I faced a similar problem, and I took a middle road.

Much of the data you will encounter will (likely) be handled the same way when you finally process it. That means much of the logic you need can be reused. The specifics include where to look for data and how to transform it into a common format. I suggest the following:

Create your MainSpider class, containing most of the logic and tasks that you need. For each site, subclass MainSpider and define logic modules as required.

main_spider.py

class MainSpider(object):
# Do things here
    def get_links(url)

        return links

spider_mysite.py

from main_spider import MainSpider
class SpiderMysite(MainSpider):
    def get_data(links):
        for link in links:

            # Do more stuff. 

Hope it helps.