粗暴的拒绝规则

Question

嘿，我目前正在使用scrapy，并且在运行爬网时注意到我的拒绝规则被完全忽略，导致多次刮擦相同的物品，任何人都可以告诉我原因。 任何帮助表示赞赏

class DIY_spider(CrawlSpider):
    name = 'diy_cat'
    allowed_domains = ['diy.com']

    start_urls =[
        #"http://www.diy.com",
        "http://www.diy.com/nav/decor",
        "http://www.diy.com/nav/garden",
        "http://www.diy.com/nav/rooms",
        "http://www.diy.com/nav/fix",
        "http://www.diy.com/nav/build",

    ]

    rules = (
        Rule(SgmlLinkExtractor(allow=(r'/nav/decor|garden|rooms|fix|build/([A-Za-z0-9-]*)$'),
                               deny=('//diy/jsp/',
                                     'pricerange',
                                     'productId',
                                     '-size%',
                                     'tab=rev')),follow=True),

        Rule(SgmlLinkExtractor(allow=(r'/nav/decor|garden|rooms|fix|build/(.*)[0-9]{8}$' ),)
             , follow=True, callback='parse_items'),

** * ** * ** * ** * ** * 编辑 * ** * ** * ** * ** * ** *

这是日志中发生的事情

2014-04-07 15：01：47 + 0100 [diy_cat]调试：从<200 http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec中删除-1200-1300 / -size％3E3_29_x_2_39m / Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712？height = 411＆mediaId = m8416757＆productId = 13538712＆skuId = 14009418＆width = 411 >

{'currency_code': 'GBP',
 'supplier_name': 'www.diy.com',
 'supplier_part_description': u'19mm tongue and groove interlocking timber to side walls, 12mm tongue and groove timber to floor and roof, supplied with pressure treated floor joists and green roofing felt.',
 'supplier_part_name': u'Shire 11x8 Berryfield Log Cabin - Home Delivered Only',
 'supplier_part_number': u'5019804112289',
 'supplier_price_gross': 1249.98,
 'supplier_price_net': 1041.65,
 'supplier_price_tax_amount': 208.33,
 'supplier_url': 'http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec-1200-1300/-size%3E3_29_x_2_39m/Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712'}

2014-04-07 15：01：47 + 0100 [diy_cat]调试：从<200 http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec中删除-1200-1300 / -size％3E3_29_x_2_39m / Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712？height = 411＆mediaId = m8416844＆productId = 13538712＆skuId = 14009418＆width = 411 >

{'currency_code': 'GBP',
 'supplier_name': 'www.diy.com',
 'supplier_part_description': u'19mm tongue and groove interlocking timber to side walls, 12mm tongue and groove timber to floor and roof, supplied with pressure treated floor joists and green roofing felt.',
 'supplier_part_name': u'Shire 11x8 Berryfield Log Cabin - Home Delivered Only',
 'supplier_part_number': u'5019804112289',
 'supplier_price_gross': 1249.98,
 'supplier_price_net': 1041.65,
 'supplier_price_tax_amount': 208.33,
 'supplier_url': 'http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec-1200-1300/-size%3E3_29_x_2_39m/Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712'}

2014-04-07 15：01：47 + 0100 [diy_cat]调试：从<200 http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec中删除-1200-1300 / -size％3E3_29_x_2_39m / Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712？height = 411＆mediaId = m8417696＆productId = 13538712＆skuId = 14009418＆width = 411 >

{'currency_code': 'GBP',
 'supplier_name': 'www.diy.com',
 'supplier_part_description': u'19mm tongue and groove interlocking timber to side walls, 12mm tongue and groove timber to floor and roof, supplied with pressure treated floor joists and green roofing felt.',
 'supplier_part_name': u'Shire 11x8 Berryfield Log Cabin - Home Delivered Only',
 'supplier_part_number': u'5019804112289',
 'supplier_price_gross': 1249.98,
 'supplier_price_net': 1041.65,
 'supplier_price_tax_amount': 208.33,
 'supplier_url': 'http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec-1200-1300/-size%3E3_29_x_2_39m/Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712'}

2014-04-07 15：01：47 + 0100 [diy_cat]调试：从<200 http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec中删除-1200-1300 / -size％3E3_29_x_2_39m / Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712？heroPopup = true＆mediaId = m8417696＆productId = 13538712＆skuId = 14009418 >

{'currency_code': 'GBP',
 'supplier_name': 'www.diy.com',
 'supplier_part_description': u'19mm tongue and groove interlocking timber to side walls, 12mm tongue and groove timber to floor and roof, supplied with pressure treated floor joists and green roofing felt.',
 'supplier_part_name': u'Shire 11x8 Berryfield Log Cabin - Home Delivered Only',
 'supplier_part_number': u'5019804112289',
 'supplier_price_gross': 1249.98,
 'supplier_price_net': 1041.65,
 'supplier_price_tax_amount': 208.33,
 'supplier_url': 'http://www.diy.com/nav/garden/garden-buildings/cabins-summerhouses/-constructiontype-Interlocking/-pricerangec-1200-1300/-size%3E3_29_x_2_39m/Shire-11x8-Berryfield-Log-Cabin-Home-Delivered-Only-13538712'}

Answer 1

测试了蜘蛛和规则之后，我相信您的拒绝规则没有正确指定（缺少一个或两个逗号），更具体地说，对于“ / diy / jsp /”的拒绝规则不太正确。

我使用以下修改后的规则将蜘蛛运行了大约10秒钟左右，并且在日志中找不到任何“ diy / jsp”实例，因此我认为这是可行的。 就是说，值得将mediaId添加到拒绝列表中，因为此URL参数是上面发布的重复URL中的唯一主要区别。

rules = (
    Rule(SgmlLinkExtractor(allow=(r'/nav/decor|garden|rooms|fix|build/([A-Za-z0-9-]*)$'),
                           deny=('\.\./\.\./diy/jsp/',
                                 'pricerange',
                                 'productId',
                                 '-size%',
                                 'tab=rev'),),follow=True),

    Rule(SgmlLinkExtractor(allow=(r'/nav/decor|garden|rooms|fix|build/(.*)[0-9]{8}$' ),)
         , follow=True, callback='parse_items'),
    )