在Scrapy中使用递归蜘蛛[Python]

Question

Intro Hello i'm using Scrapy in order to take data in Yahoo Answer. My achievement is to take all the question and the answer in one precise section.

I try Using scrapy and selenium first of all i try to take a list of any question in a section, this list is memorize in the Spider Class. After i use a For loop in order to parse every single page.

 for url in self.start_urls_mod:
        yield scrapy.Request(url, callback=self.parse_page)
        i = i + 1

the method parse_page is structured for scrapying the question page, the best answer and all the other answer. This works fine.

The problem comes when i try to go on "Next" question using the href in the "next" link present on following link on the right side of the page. I call again the same function parse_page, passing the url take from that link. Sometimes this work but other times no. I don't now if is correct to call two times the parse_page function, without using any base case in other to stop the recursion its stop anyway.

The program works without any error and stops, but i don't find any question in the "next" section. Only someone.

There is a snippet of my code.

    def parse_page(self, response):
    #Scraping with xpath things that interests me
    #Go to the next similar question
    next_page = hxs.xpath('((//a[contains(@class,"Clr-b")])[3])/@href').extract()
    composed_string = "https://answers.yahoo.com" + next_page[0]
    print("NEXT -> "+str(composed_string))
    yield scrapy.Request(urljoin(response.url, composed_string), callback=self.parse_page)

ps. I would use a crowl spider, but i can't define any rules to take only this type of question. So please how i can improve my function.

Infos: https://answers.yahoo.com/question/index?qid=20151008101821AAuHgCk

Answer 1

First of all your XPath for selecting the next URL is wrong. You will always obtain the third URL with "Clr-b" which can be wrong (it does not exist or it is not the next site).

For such queries I would use the text search. In your case something like this:

next_page = response.xpath('//a[contains(@class,"Clr-b") and text()=" Next "]/@href').extract()

Then you compose your URL as you do and you do not have to use urljoin . This is not needed because you already have the right URL which you need to yield as you do. This is probably the cause why your spider stops: you generate a URL with urljoin which is not found -- and this is not the URL you print to the console.

And it is no problem to use the same function as callback.