Scrapy Spiders-處理非HTML鏈接（PDF，PPT等。）

Question

我正在學習Scrapy和Python，並從一個空白項目開始。 我正在使用Scrapy LxmlLinkExtractor解析鏈接，但是在遇到非HTML鏈接/頁面（例如PDfs或其他文檔）時，蜘蛛始終會卡住。

問題：如果我只想存儲那些URls，我們通常如何處理與Scrapy的那些鏈接（我現在暫時不希望文檔的內容...）

包含文檔的示例頁面： http : //afcorfmc.org/2009.html

這是我的蜘蛛代碼：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.lxmlhtml import LxmlLinkExtractor
from super.items import SuperItem
from scrapy.selector import Selector

class mySuper(CrawlSpider):
    name="super"
    #on autorise seulement le crawl du site indiqué dans allowed_domains
    allowed_domains = ['afcorfmc.org']

    #démarrage sur la page d'accueil du site
    start_urls = ['http://afcorfmc.org']

    rules = (Rule (LxmlLinkExtractor(allow=(),deny=(),restrict_xpaths=()), callback="parse_o", follow= True),)

    def parse_o(self,response):
        #récupération des datas récoltées (contenu de la page)
        sel = Selector(response)

        #on prépare item, on va le remplir (souvenez-vous, dans items.py)
        item = SuperItem()

        #on stocke l'url de la page dans le tableau item
        item['url'] = response.url

        #on récupère le titre de la page ( titre ) grâce à un chemin xpath
        #item['titre'] = sel.xpath('//title/text()').extract()

        # on fait passer item à la suite du processus
        yield item

Answer 1

如scrapy LinkExtractor文檔中所述 ， LxmlLinkExtractor默認情況下不包括具有某些擴展名的鏈接：請參閱https://github.com/scrapy/scrapy/blob/master/scrapy/linkextractors/ init .py＃L20

擴展名列表包括.pdf ， .ppt 。

您可以將deny_extensions參數添加到LxmlLinkExtractor實例，並將其留空，例如：

$ scrapy shell http://afcorfmc.org/2009.html
2014-10-27 10:27:02+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
...
2014-10-27 10:27:03+0100 [default] DEBUG: Crawled (200) <GET http://afcorfmc.org/2009.html> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f5b1a6f4910>
[s]   item       {}
[s]   request    <GET http://afcorfmc.org/2009.html>
[s]   response   <200 http://afcorfmc.org/2009.html>
[s]   settings   <scrapy.settings.Settings object at 0x7f5b2013f450>
[s]   spider     <Spider 'default' at 0x7f5b19e9bed0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

In [1]: from scrapy.contrib.linkextractors.lxmlhtml import LxmlLinkExtractor

In [2]: lx = LxmlLinkExtractor(allow=(),deny=(),restrict_xpaths=(), deny_extensions=())

In [3]: lx.extract_links(response)
Out[3]: 
[Link(url='http://afcorfmc.org/documents/TOPOS/2009/MARS/ANATOMO_PATHOLOGIE_Dr_Guinebretiere.ppt', text='ANATOMO_PATHOLOGIE_Dr_Guinebretiere.ppt', fragment='', nofollow=False),
 Link(url='http://afcorfmc.org/documents/TOPOS/2009/MARS/CHIMIOTHERAPIE_Dr_Toledano.ppt', text='CHIMIOTHERAPIE_Dr_Toledano.ppt', fragment='', nofollow=False),
 Link(url='http://afcorfmc.org/documents/TOPOS/2009/MARS/CHIRURGIE_Dr_Guglielmina.ppt', text='CHIRURGIE_Dr_Guglielmina.ppt', fragment='', nofollow=False),
 Link(url='http://afcorfmc.org/documents/TOPOS/2009/MARS/CHIRURGIE_Dr_Sebban.ppt', text='CHIRURGIE_Dr_Sebban.ppt', fragment='', nofollow=False),
 Link(url='http://afcorfmc.org/documents/TOPOS/2009/MARS/Cas_clinique_oesophage.ppt', text='Cas_clinique_oesophage.ppt', fragment='', nofollow=False),
 Link(url='http://afcorfmc.org/documents/TOPOS/2009/MARS/IMAGERIE_Dr_Seror.ppt', text='IMAGERIE_Dr_Seror.ppt', fragment='', nofollow=False),
 ...
 Link(url='http://afcorfmc.org/documents/TOPOS/2009/OCTOBRE/VB4_Technique%20monoisocentrique%20dans%20le%20sein%20Vero%20Avignon%202009.pdf', text='VB4_Technique monoisocentrique dans le sein Vero Avignon 2009.pdf', fragment='', nofollow=False)]

In [4]: