过滤刮擦管道中的物料

Question

I've scrapped the urls i want from a page. Now I want to filter them for keywords using a pipeline:

class GumtreeCouchesPipeline(object):

keywords = ['leather', 'couches']

def process_item(self, item, spider):
    if any(key in item['url'] for key in keywords):
        return item

Problem is its returning nothing now.

The spider:

import scrapy
from gumtree_couches.items import adItem
from urllib.parse import urljoin

class GumtreeSpider(scrapy.Spider):
    name = 'GumtreeCouches'
    allowed_domains = ['https://someurl']
    start_urls = ['https://someurl']


def parse(self, response):
    item = adItem()
    for ad_links in response.xpath('//div[@class="view"][1]//a'):
        relative_url = ad_links.xpath('@href').extract_first()
        item['title'] = ad_links.xpath('text()').extract_first()
        item['url'] = response.urljoin(relative_url)

        yield item

How can I filter all the scraped urls for keywords using the pipeline? Thanks!

Answer 1

This should fix your problem:

class GumtreeCouchesPipeline(object):

    keywords = ['leather', 'couches']

    def process_item(self, item, spider):
        if any(key in item['url'] for key in self.keywords):
            return item

Notice that I'm using self.keywords to refer to the keywords class attribute.

If you look at your spider logs, you should find some errors saying something like: NameError: name 'keywords' is not defined .

Anyway, I'd recommend you to implement this pipeline like this:

from scrapy.exceptions import DropItem

class GumtreeCouchesPipeline(object):

    keywords = ['leather', 'couches']

    def process_item(self, item, spider):
        if not any(key in item['url'] for key in self.keywords):
            raise DropItem('missing keyword in URL')
        return item

This way, you'll have the information about the dropped items in the job stats once it's finished.

Answer 2

From reading the documentation I think you have to cater for all paths eg

from scrapy.exceptions import DropItem

    def process_item(self, item, spider):
        keywords = ['leather', 'couches']
        if item['url']:
            if any(key in item['url'] for key in keywords):
                return item
            else
                raise DropItem("Missing specified keywords.")
        else
            return item