簡單的Scrapy爬蟲不關注鏈接和抓取

Question

基本上問題在於鏈接

我將從第1..2..3..4..5 .....頁共90頁

每頁有100個左右的鏈接

每頁都是這種格式

http://www.consumercomplaints.in/lastcompanieslist/page/1
http://www.consumercomplaints.in/lastcompanieslist/page/2
http://www.consumercomplaints.in/lastcompanieslist/page/3
http://www.consumercomplaints.in/lastcompanieslist/page/4

這是正則表達式匹配規則

Rule(LinkExtractor(allow='(http:\/\/www\.consumercomplaints\.in\/lastcompanieslist\/page\/\d+)'),follow=True,callback="parse_data")

我要轉到每個頁面，然后創建一個Request對象以刮取每個頁面中的所有鏈接

Scrapy每次總共僅抓取179個鏈接，然后給出finished狀態

我究竟做錯了什么？

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
import urlparse

class consumercomplaints_spider(CrawlSpider):
    name = "test_complaints"
    allowed_domains = ["www.consumercomplaints.in"]
    protocol='http://'

    start_urls = [
        "http://www.consumercomplaints.in/lastcompanieslist/"
    ]

    #These are the rules for matching the domain links using a regularexpression, only matched links are crawled
    rules = [
        Rule(LinkExtractor(allow='(http:\/\/www\.consumercomplaints\.in\/lastcompanieslist\/page\/\d+)'),follow=True,callback="parse_data")
    ]


    def parse_data(self, response):
        #Get All the links in the page using xpath selector
        all_page_links = response.xpath('//td[@class="compl-text"]/a/@href').extract()

        #Convert each Relative page link to Absolute page link -> /abc.html -> www.domain.com/abc.html and then send Request object
        for relative_link in all_page_links:
            print "relative link procesed:"+relative_link

            absolute_link = urlparse.urljoin(self.protocol+self.allowed_domains[0],relative_link.strip())
            request = scrapy.Request(absolute_link,
                         callback=self.parse_complaint_page)
            return request


        return {}

    def parse_complaint_page(self,response):
        print "SCRAPED"+response.url
        return {}

Answer 1

您將需要使用yield而不是return。

對於每個新的Request對象，請使用yield request而不是return reqeust

查看更多有關產量在這里和他們與理性之間的差異在這里