Scrapy start_urls are not parsed

Question

I just updated scrapy to 0.22 and are facing a couple of problems now.

I'm using a the basic Spider and just simply want to request one start_url and see a result in the parse(self,response) function.

class wildberries(Spider):

    name = "wildberries"
    allowed_domains = ['wildberries.ru']
    start_urls = ['http://www.wildberries.ru/']

    def parse(self, response):
        hxs         = HtmlXPathSelector(response)
        print "Hello World", response.url
        maincats    = hxs.select('//div[@class="tomenu"]/ul[@class="green_menu"]/li/a/@href').extract()
        print maincats

The only output I get from executing the spider is:

[jdargel@rhel1 krauler]$ scrapy crawl wildberries
2014-02-25 12:46:19+0400 [scrapy] INFO: Scrapy 0.22.2 started (bot: scrapybot)
2014-02-25 12:46:19+0400 [scrapy] INFO: Optional features available: ssl, http11
2014-02-25 12:46:19+0400 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE':       'krauler.spiders', 'LOG_STDOUT': True, 'SPIDER_MODULES': ['krauler.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64)', 'CONCURRENT_REQUESTS': 1}

I don't know why my start_url is not parsed and since I don't get any response from parse() I assume that the function is not even called. Would be glad for suggestions and if any questions occur, I'm more than happy to provide further code.

Answer 1

Indentation looks weird. I don't know if it's because of StackOverflow or something, but when trying out

from scrapy.spider import Spider
from scrapy.selector import Selector


class wildberries(Spider):
    name = "wildberries"
    allowed_domains = ['wildberries.ru']
    start_urls      = ['http://www.wildberries.ru/']

    def parse(self, response):
        selector = Selector(response)
        print "Hello World", response.url
        maincats = selector.xpath('//div[@class="topmenu"]/ul[@class="green_menu"]/li/a/@href').extract()
        print maincats

I do get "correct" output:

paul@wheezy:~/tmp$ scrapy runspider wildberries.py 
2014-02-25 10:34:44+0100 [scrapy] INFO: Scrapy 0.23.0 started (bot: scrapybot)
2014-02-25 10:34:44+0100 [scrapy] INFO: Optional features available: ssl, http11, boto, django
2014-02-25 10:34:44+0100 [scrapy] INFO: Overridden settings: {}
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled item pipelines: 
2014-02-25 10:34:44+0100 [wildberries] INFO: Spider opened
2014-02-25 10:34:44+0100 [wildberries] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-02-25 10:34:44+0100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2014-02-25 10:34:44+0100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-02-25 10:34:44+0100 [wildberries] DEBUG: Crawled (200) <GET http://www.wildberries.ru/> (referer: None)
Hello World http://www.wildberries.ru/
[u'http://www.wildberries.ru/', u'http://www.wildberries.ru/catalog/1/women.aspx', u'http://www.wildberries.ru/catalog/2/men.aspx', u'http://www.wildberries.ru/catalog/3/child.aspx', u'http://www.wildberries.ru/catalog/413/women.aspx', u'http://www.wildberries.ru/catalog/492/women.aspx', u'http://www.wildberries.ru/catalog/524/women.aspx', u'http://www.wildberries.ru/catalog/2467/page.aspx', u'http://www.wildberries.ru/catalog/4263/page.aspx', u'http://www.wildberries.ru/catalog/181/women.aspx']
2014-02-25 10:34:44+0100 [wildberries] INFO: Closing spider (finished)
2014-02-25 10:34:44+0100 [wildberries] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 217,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 15623,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 2, 25, 9, 34, 44, 519186),
     'log_count/DEBUG': 3,
     'log_count/INFO': 7,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2014, 2, 25, 9, 34, 44, 143133)}
2014-02-25 10:34:44+0100 [wildberries] INFO: Spider closed (finished)