获得关注链接的最佳方法抓取网络抓取工具

Question

So I'm trying to write a spider to continue clicking a next button on a webpage until it can't anymore (or until I add some logic to make it stop). The code below correctly gets the link to the next page but prints it only once. My question is why isn't it "following" the links that each next button leads to?

class MyprojectSpider(scrapy.Spider):
    name = 'redditbot'
    allowed_domains = ['https://www.reddit.com/r/nfl/?count=25&after=t3_7ax8lb']
    start_urls = ['https://www.reddit.com/r/nfl/?count=25&after=t3_7ax8lb']

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        next_page = hxs.select('//div[@class="nav-buttons"]//a/@href').extract()
        if next_page:
            yield Request(next_page[1], self.parse)
            print(next_page[1])

Answer 1

To go to the next page, instead of printing the link you just need to yield a scrapy.Request object like the following code:

import scrapy

class MyprojectSpider(scrapy.Spider):
    name = 'myproject'
    allowed_domains = ['reddit.com']
    start_urls = ['https://www.reddit.com/r/nfl/']

    def parse(self, response):
        posts = response.xpath('//div[@class="top-matter"]')
        for post in posts:
            # Get your data here
            title = post.xpath('p[@class="title"]/a/text()').extract()
            print(title)
            # Go to next page
            next_page = response.xpath('//span[@class="next-button"]/a/@href').extract_first()
            if next_page:
                 yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

Update: Previous code was wrong, needed to use the absolute URL and also some Xpaths were wrong, this new one should work.

Hope it helps!