Scrapy：在没有自定义 Feed Exporter 的情况下自定义 Item Pipeline 后使用 Feed Exports class？

Question

My Spider looks like this:

class ExampleSpider(scrapy.Spider):
    name = 'example'

    custom_settings = {
        'ITEM_PIPELINES': {'img_clear.pipelines.DuplicatesPipeline': 100,},
        'FEEDS': {
            'feeds/example/tags.csv': {
                'format': 'csv',
                'fields': ["tag_id", "url", "title"],
                'item_export_kwargs': {
                    'include_headers_line': False,
                },
                'item_classes': [ExampleTagItem],
                'overwrite': False
            },
            'feeds/example/galleries.csv': {
                'format': 'csv',
                'fields': ["id", "url", "tag_ids"],
                'item_export_kwargs': {
                    'include_headers_line': False,
                },
                'item_classes': [ExampleGalleryItem],
                'overwrite': False,
            }
        }
    }

This is the img_clear.pipelines.DuplicatesPipeline :

class DuplicatesPipeline():
    def open_spider(self, spider):
        if spider.name == "example":
            with open("feeds/example/galleries.csv", "r") as rf:
                csv = rf.readlines()
            self.ids_seen = set([str(line.split(",")[0]) for line in csv])
            
            with open("feeds/example/tags.csv", "r") as rf:
                tags_csv = rf.readlines()
            self.tag_ids_seen = set([str(line.split(",")[0]) for line in tags_csv])

    def process_item(self, item, spider):
        if isinstance(item, ExampleTagItem):
            self.process_example_tag_item(item, spider)    
        elif isinstance(item, ExampleGalleryItem):
            self.process_example_gallery_item(item, spider)

    def process_example_tag_item(self, item, spider):
        adapter = ItemAdapter(item)
        if adapter['tag_id'] in self.tag_ids_seen:
            raise DropItem(f"Duplicate item found: {item!r}")
        else:
            self.tag_ids_seen.add(adapter['tag_id'])
            return item

    def process_example_gallery_item(self, item, spider):
        adapter = ItemAdapter(item)
        if adapter['id'] in self.ids_seen:
            raise DropItem(f"Duplicate item found: {item!r}")
        else:
            self.ids_seen.add(adapter['id'])
            return item

With the item pipeline activated it will drop some items (logging: [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'tag_id': '4',... ) and return others (logging: [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.example.com/10232335/> ) but noting is written to the files. Somehow the returned items don't seem to reach the feed exports extension. What am I missing?

• When commenting out the 'ITEM_PIPELINES': {'img_clear.pipelines.DuplicatesPipeline': 100,}, in the custom_settings , items are saved in the right csv-files.
• Using scrapy crawl example -o test.csv will create an empty csv when the pipeline is activated as well. So it seems that the issue is with the pipeline.
• Printing the items right before they should be returned did print correct item information
• The pipeline is derived from the scrapy docs .

Answer 1

I have come across this same issue many times as well. It has something to do with the path used in the FEEDS section of your custom settings. There are a few things that you can try to fix the problem.

1. Using absolute paths.

On linux/mac:

'FEEDS': {
    '/absolute/path/to/feeds/example/galleries.csv': {
    ...
}

on Windows you can try it a few ways:

'FEEDS': {
    r'C:\Users\path\to\feeds\example\galleries.csv': {
        ...
}

or also on windows

'FEEDS': {
    '/Users/path/to/feeds/example/galleries.csv': {
        ...
}

2. Adding a schema to the FEEDS ( 'file///' ) path always seems to work even though it is mentioned in the documentation that it isn't necessary when exporting to local filesystem.

'FEEDS': {
    'file:///absolute/path/to/feeds/example/galleries.csv': {
        ...
}

Answer 2

Thanks for the response, I'm not sure if this would actually have fixed it. since the feed was working perfectly with relative paths when the pipeline is deactivated. I might test that anyways some time.

However, I figured out an other mistake in my code that fixed it without changing the paths: The docs state, that the process_item function must return an item object , return a twisted Deferred or raise a DropItem exception. My code was derived from here but I missed the return statements in the lines calling the process_..._item functions.

Tbh, I discovered the solution by accident trying to replicate my issue in a less complex spider and wrote up something like this and it worked:

def process_item(self, item, spider):
    if isinstance(item, ExampleTagItem):
        adapter = ItemAdapter(item)
        if adapter['tag_id'] in self.tag_ids_seen:
            raise DropItem(f"Duplicate item found: {item!r}")
        else:
            self.tag_ids_seen.add(adapter['tag_id'])
        return item
    elif isinstance(item, ExampleGalleryItem):
        adapter = ItemAdapter(item)
        if adapter['id'] in self.ids_seen:
            raise DropItem(f"Duplicate item found: {item!r}")
        else:
            self.ids_seen.add(adapter['id'])
        return item

Since I'm very new to coding: Any suggestions how to reduce the repetition in this code? I could use "id" in both Item objects but still would need to differentiate between the two sets so no idea how to do this...