Python Scrapy仅爬网start_urls然后停止。 如何更深入？

Question

Why is it that Scrapy only crawls the start_urls and then stops? Is there a way to have Scrapy crawl through all pages in a directory tree of a website, such as http://www.example.com/directory ? Or, is there a way to have Scrapy crawl deeper into all links on the start_urls pages?

class DmozSpider(CrawlSpider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
                      "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
                      "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
                      ]

        rules = [
               Rule(SgmlLinkExtractor(allow=('', )), follow=True),
               Rule(SgmlLinkExtractor(allow=('', )), callback='parse_item')
               ]

        def parse_item(self, response):
          print response.url

        def parse(self, response):
          print response.url

Here's the code in my main.py file:

dmozSpider = DmozSpider()
spider = dmozSpider

settings = get_project_settings()

crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()

Answer 1

在DmozSpider类中删除parse（），然后parse_item（）将获得比start_urls更多的信息

Answer 2

To elaborate on @stevetronix's answer a bit:

You are not supposed to override the parse() method when using the CrawlSpider . You should set a custom callback in your Rule with a different name.
Here is the excerpt from the official documentation :

When writing crawl spider rules, avoid using parse as callback, since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work.