递归刮后没有刮数据

Question

I'm trying to recursively scrape the titles of jobs from https://iowacity.craigslist.org/search/jjj . That is to say, I want the spider to scrape all the job titles on Page 1 and then follow the link "next>" on the bottom to scrape the next page, and so on. I mimicked Michael Herman's tutorial to write my spider. http://mherman.org/blog/2012/11/08/recursively-scraping-web-pages-with-scrapy/#.ViJ6rPmrTIU .

Here is my code:

import scrapy
from craig_rec.items import CraigRecItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor


class CraigslistSpider(CrawlSpider):
    name = "craig_rec"
    allowed_domains = ["https://craigslist.org"]
    start_urls = ["https://iowacity.craigslist.org/search/jjj"]

    rules = (
        Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

    def parse_items(self, response):
        items = []
        for sel in response.xpath("//span[@class = 'pl']"):
            item = CraigRecItem()
            item['title'] = sel.xpath("a/text()").extract()
            items.append(item)
        return items  

I released the spider but no data was scraped. Any help? Thanks!

Answer 1

When you set your allowed_domains to " https://craigslist.org " it stops crawling due to offsite request to subdomain 'iowacity.craigslist.org'.

You must set it as:

allowed_domains = ["craigslist.org"]

According to the docs allowed_domains is a list of strings containing domains that this spider is allowed to crawl. It expects it to be in the format domain.com , which allows the domain itself and all the subdomains to be parsed by the spider.

You can also be specific allowing only few subdomains or allow all requests by leaving the attribute empty.

Answer 2

Michael Herman's tutorial is great, but for an older version of scrapy. This snippet avoids some deprecation warnings, and also turns parse_page into a generator:

import scrapy
from craig_rec.items import CraigRecItem
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class CraiglistSpider(CrawlSpider):
    name = "craiglist"
    allowed_domains = ["craigslist.org"]
    start_urls = (
        'https://iowacity.craigslist.org/search/jjj/',
    )

    rules = (
        Rule(LinkExtractor(restrict_xpaths=('//a[@class="button next"]',)),
             callback="parse_page", follow=True),
    )

    def parse_page(self, response):
        for sel in response.xpath("//span[@class = 'pl']"):
            item = CraigRecItem()
            item['title'] = sel.xpath(".//a/text()").extract()
            yield item

This post also has some great tips on scraping Craigslist.