[英]Cannot download images from website with scrapy
我從Scrapy開始,以便自動從網站下載文件。 作為測試,我想從該網站下載jpg文件。 我的代碼基於Scrapy網站上的入門教程和“ 文件和圖像管道”教程 。
我的代碼是這樣的:
在settings.py中,我添加了以下幾行:
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '/home/lucho/Scrapy/jpg/'
我的items.py文件是:
import scrapy
class JpgItem(scrapy.Item):
image_urls = scrapy.Field()
images = scrapy.Field()
pass
我的管道文件是:
import scrapy
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
class JpgPipeline(object):
def process_item(self, item, spider):
return item
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield scrapy.Request(image_url)
def item_completed(self, results, item, info):
image_paths = [x['path'] for ok, x in results if ok]
if not image_paths:
raise DropItem("Item contains no images")
item['image_paths'] = image_paths
return item
最后,我的蜘蛛文件是:
import scrapy
from .. items import JpgItem
class JpgSpider(scrapy.Spider):
name = "jpg"
allowed_domains = ["http://www.kevinsmedia.com"]
start_urls = [
"http://www.kevinsmedia.com/km/mp3z/Fluke/Risotto/"
]
def init_request(self):
#"""This function is called before crawling starts."""
return Request(url=self.login_page, callback=self.parse)
def parse(self, response):
item = JpgItem()
return item
( 理想情況下,我要下載所有jpg,而不為每個所需文件指定確切的網址 )
“ scrapy crawl jpg”的輸出為:
2015-12-08 19:19:30 [scrapy] INFO: Scrapy 1.0.3.post6+g2d688cd started (bot: jpg)
2015-12-08 19:19:30 [scrapy] INFO: Optional features available: ssl, http11
2015-12-08 19:19:30 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'jpg.spiders', 'SPIDER_MODULES': ['jpg.spiders'], 'COOKIES_ENABLED': False, 'DOWNLOAD_DELAY': 3, 'BOT_NAME': 'jpg'}
2015-12-08 19:19:30 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2015-12-08 19:19:30 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-12-08 19:19:30 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-12-08 19:19:30 [scrapy] INFO: Enabled item pipelines: ImagesPipeline
2015-12-08 19:19:30 [scrapy] INFO: Spider opened
2015-12-08 19:19:30 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-12-08 19:19:30 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-12-08 19:19:31 [scrapy] DEBUG: Crawled (200) <GET http://www.kevinsmedia.com/km/mp3z/Fluke/Risotto/> (referer: None)
2015-12-08 19:19:31 [scrapy] DEBUG: Scraped from <200 http://www.kevinsmedia.com/km/mp3z/Fluke/Risotto/>
{'images': []}
2015-12-08 19:19:31 [scrapy] INFO: Closing spider (finished)
2015-12-08 19:19:31 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 254,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 2975,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 12, 8, 22, 19, 31, 294139),
'item_scraped_count': 1,
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2015, 12, 8, 22, 19, 30, 619918)}
2015-12-08 19:19:31 [scrapy] INFO: Spider closed (finished)
盡管似乎沒有錯誤,但程序未檢索jpg文件。 萬一重要,我正在使用Ubuntu。
您尚未在JpgSpider
類中定義parse()
。
更新。 這看起來不像是問題,您應該立即抓狂地攻擊,因為我可以在更新中看到該URL。 WGET可能更合適,請在此處查看答案 。 特別是,請查看頂部答案的第一個注釋,以了解如何使用文件擴展名限制下載的文件( -A jpg
)。
更新2:parse()例程可以使用此代碼從<a>
標記獲取專輯插圖的URL
part_urls = response.xpath('//a[contains(., "AlbumArt")]/@href')
這將返回部分URL的列表,您需要從response.url
添加要解析的頁面的根URL。 我查看過的網址中有一些%代碼,它們可能是個問題,但還是請嘗試一下。 獲得完整URL的列表后,將其放入item []
item['image_urls'] = full_urls
yield item
自動下載圖像應該很麻煩,因此您可以刪除中間件,讓scrapy做繁重的工作。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.