Scrapy搜寻器中的URL不会提供给下一个解析器

Question

I came across a yielding problem, when I was trying to crawl http://www.brand-in-trend.ru . As you see below, I'm using Scrapy and defined a Basespider. The first parser works perfectly fine and returns all brands found on the start_url.

Now, when I want to yield the callback Request to the categories parser, I don't get a response nor an Error. The spider just quits.

Spider:

class brandintrend(BaseSpider):
name = "brandintrend"

allowed_domains = [ 'trend-in-brand.ru' ]

start_urls      = [ 'http://brand-in-trend.ru/brands/' ]

def parse(self, response):
    hxs         = HtmlXPathSelector(response)
    brands      = hxs.select('//div[@class="brandcol"]/ul/li/a/@href').extract()

    for brand in brands:
        brand = "http://www.brand-in-trend.ru" + brand
        print brand
        # request = Request(brand, callback=self.categories)
        yield Request(brand, callback=self.categories)

def categories(self, response):
    print "Hello World"
    hxs = HtmlXPathSelector(response)
    print response.url

I tried the following already to solve this issue:

1. I tested the generated brand urls (ex. http://www.brand-in-trend.ru/brands/parker/ ) in Chrome (Javasript turned off) and they worked fine.
2. I put all generated brand urls in the start_url list and tried to yield those directly to the categories parser
3. I looked at this post, which unfortunately didn't solve my problem: scrapy unable to make Request() callback

If anybody came across a similar problem I would be greatful for a solution or advise

Thanks in advance

J

Answer 1

This is because you set:

allowed_domains = [ 'trend-in-brand.ru' ]

but, you are crawling the url from a different domain:

start_urls = [ 'http://brand-in-trend.ru/brands/' ]

See trend-in-brand vs brand-in-trend .