Scrapy Crawl Spider没有关注链接

Question

So i wrote a web crawler to extract food items from walmart.com. Here is my spider. I cant seem to figure out why it does not follow the links on the left until. It pulls the main page then finishes.

My intended goal is for it to follow all the links on the left flyout bar then exract each food item from those pages.

I even tried just using allow=() so that it follows every link on the page but that still does not work.

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.loader import XPathItemLoader
from scrapy.contrib.loader.processor import Join, MapCompose
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor as sle
from walmart_scraper.items import GroceryItem


class WalmartFoodSpider(CrawlSpider):
    name = "walmart_scraper"
    allowed_domains = ["www.walmart.com"]
    start_urls = ["http://www.walmart.com/cp/976759"]
    rules = (Rule(sle(restrict_xpaths=('//div[@class="lhn-menu-flyout-inner lhn-menu-flyout-2col"]/ul[@class="block-list"]/li/a',)),callback='parse',follow=True),)

    items_list_xpath = '//div[@class="js-tile tile-grid-unit"]'

item_fields = {'title': './/a[@class="js-product-title"]/h3[@class="tile-heading"]/div',
               'image_url': './/a[@class="js-product-image"]/img[@class="product-image"]/@src',
               'price': './/div[@class="tile-price"]/div[@class="item-price-            container"]/span[@class="price price-display"]|//div[@class="tile-price"]/div[@class="item-price-   container"]/span[@class="price price-display price-not-available"]',
               'category': '//nav[@id="breadcrumb-container"]/ol[@class="breadcrumb-list"]/li[@class="js-breadcrumb breadcrumb "][2]/a',
               'subcategory': '//nav[@id="breadcrumb-container"]/ol[@class="breadcrumb-list"]/li[@class="js-breadcrumb breadcrumb active"]/a',
               'url': './/a[@class="js-product-image"]/@href'}
def parse(self, response):

    selector = HtmlXPathSelector(response)

    # iterate over deals
    for item in selector.select(self.items_list_xpath):
        loader = XPathItemLoader(GroceryItem(), selector=item)

        # define processors
        loader.default_input_processor = MapCompose(unicode.strip)
        loader.default_output_processor = Join()

        # iterate over fields and add xpaths to the loader
        for field, xpath in self.item_fields.iteritems():
            loader.add_xpath(field, xpath)
        yield loader.load_item()

Answer 1

You are not supposed to override the parse() method when using the CrawlSpider . You should set a custom callback in your Rule with a different name.
Here is the excerpt from the official documentation :

When writing crawl spider rules, avoid using parse as callback, since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work.