使用scrapy从网站获取链接

Question

i am trying to extract the links from one class and store it using scrapy. I am not really sure what's the problem. Here is the code:

import scrapy

from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["craigslist.org"]
    start_urls = [
    "http://losangeles.craigslist.org/search/jjj"
]

def parse(self, response):
    for sel in response.xpath('//a[@class="hdrlnk"]'):
        item = DmozItem()
        item['link'] = sel.xpath('//a/@href').extract()

        yield item

CMD line

scrapy crawl dmoz -o items.csv -t csv

Any help is very appreciated, thanks in advance!

Answer 1

I have updated the code with few things which were missing Check out:

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import Selector



class CompItem(scrapy.Item):
    link = scrapy.Field()




class criticspider(CrawlSpider):
    name = "craig"
    allowed_domains = ["losangeles.craigslist.org"]
    start_urls = ["http://losangeles.craigslist.org/search/jjj"]


    def parse_start_url(self, response):
        sites = response.xpath('//div[@class="content"]')
        items = []

        for site in sites:
            item = CompItem()
            item['link'] = site.xpath('.//p[@class="row"]/span/span[@class="pl"]/a/@href').extract()
            items.append(item)
            return items

Answer 2

If you are getting some error like the following

exceptions.NotImplementedError:

seems like your parse() function is not intended properly.

I have slightly modified your code

# -*- coding: utf-8 -*-
import scrapy

# item class included here 
class DmozItem(scrapy.Item):
    # define the fields for your item here like:
    link = scrapy.Field()


class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["craigslist.org"]
    start_urls = [
    "http://losangeles.craigslist.org/search/jjj"
    ]

    BASE_URL = 'http://losangeles.craigslist.org'

    def parse(self, response):
        links = response.xpath('//a[@class="hdrlnk"]/@href').extract()
        for link in links:
            absolute_url = self.BASE_URL + link
            item = DmozItem(link=absolute_url)
            yield item

this will give you only the first 100 results

a sample out will be,

{'link': u'http://losangeles.craigslist.org/wst/web/5011899759.html'}