python scrapy：蜘蛛会跟踪链接，但不会下载图像

Question

I've built a basic a crawlspider to scrape the comic images from xkcd and follow links to each comic and continue scraping. The spider follows links just fine but I'm having trouble actually scraping the image.

I've tried multiple xpath and css selectors and ways of writing the parse_item method but I'm either getting errors due to scrapy trying to use the first letter of the url as the full url, or unhashable type 'list' errors and have run out of ideas.

Spider:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class XkcdSpider(CrawlSpider):
    name = 'xkcd'
    allowed_domains = ['xkcd.com']
    start_urls = ['http://xkcd.com/']

    rules = (
    Rule(LinkExtractor(allow=r'\/\d{4}\/', unique=True),
         callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        i = {}
        relative_url = response.xpath(
            '//*[@id="comic"]/img/@src').extract_first()

        absolute_url = response.urljoin(relative_url)
        i['image_urls'] = absolute_url
        return i

Items:

import scrapy


class XkcdItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    images = scrapy.Field()
    image_urls = scrapy.Field()

Image pipeline is set up like this:

ITEM_PIPELINES = {
   'scrapy.pipelines.images.ImagesPipeline': 1,
}

Traceback is either this:

TypeError: unhashable type: 'list'

Or this:

ValueError: Missing scheme in request url: h

Which I understand is from scrapy trying to use the first letter of the url rather than the whole thing but I can't find a way to make it work, have tried just .extract() rather than extract_first() but that doesn't work.

Any help greatly appreciated

Answer 1

Try it like this

srcs = response.xpath('//*[@id="comic"]/img/@src').extract()
i['image_urls'] = [response.urljoin(src) for src in srcs]

Probably you've already have done this but, just in case, be sure to set correctly the IMAGES_STORE setting.

Answer 2

Upon asking OP about the output of absolute_url he replied.

[root] INFO: imgs.xkcd.com/comics/state_borders.png Does this look right? – 

This is incorrect, that is what scraper is telling you, Missing scheme in request url: means your URL is missing the HTTP scheme information.

Also provide a list to dsads variable.

i['image_urls'] = ["https://" + absolute_url] #adding scheme to URL