从蜘蛛抓取不同的网站（Scrapy）将数据放入 CSV 文件中

Question

In an attempt to combine 2 different Scrapy spiders, that scrape unrelated websites, I created this script. But now I can't seem to put the data into a normal csv or json file. Before I combined the spiders I would just 'scrapy crawl afg2 -o data_set.csv', but now that doesnt seem to work.

What would be the easiest way to still get the data in a csv file? here is my code:

import scrapy
from scrapy.crawler import CrawlerProcess


class KhaamaSpider1(scrapy.Spider):
    name = 'khaama1'
    allowed_domains = ['www.khaama.com/category/afghanistan']
    start_urls = ['https://www.khaama.com/category/afghanistan']

    def parse(self, response):
        container = response.xpath("//div[@class='post-area']")
        for x in container:
            doc = x.xpath(".//div[@class='blog-author']/descendant::node()[4]").get()
            title = x.xpath(".//div[@class='blog-title']/h3/a/text()").get()
            author = x.xpath(".//div[@class='blog-author']/a/text()").get()
            rel_url = x.xpath(".//div[@class='blog-title']/h3/a/@href").get()

            yield{
                'date_of_creation' : doc,
                'title' : title,
                'author' : author,
                'rel_url' : rel_url
            }

class PajhwokSpider1(scrapy.Spider):
    name = 'pajhwok1'
    allowed_domains = ['www.pajhwok.com']
    start_urls = ['https://www.pajhwok.com/en/security-crime']

    def parse(self, response):
        container = response.xpath("//div[@class='node-inner clearfix']")
        for x in container:
            doc = x.xpath(".//div[@class='journlist-creation-article']/descendant::div[5]/text()").get()
            title = x.xpath(".//h2[@class='node-title']/a/text()").get()
            author = x.xpath(".//div[@class='field-item even']/a/text()").get()
            rel_url = x.xpath(".//h2[@class='node-title']/a/@href").get()

            yield{
                'date_of_creation' : doc,
                'title' : title,
                'author' : author,
                'rel_url' : rel_url
            }
        
process = CrawlerProcess()
process.crawl(KhaamaSpider1)
process.crawl(PajhwokSpider1)
process.start()

Answer 1

Usually, we have one spider to crawl one website and we can save the output in the file accordingly. As you are using two spiders for scrape to different sites in a single file an option might help to write a pipeline and store data according to into the file according to spider name name = 'pajhwok1' Have a look at this link https://www.tutorialspoint.com/scrapy/scrapy_item_pipeline.htm

Answer 2

example pipeliine.py for 2 spiders. It will close json file after 2nd spider. You can gate more information here https://docs.scrapy.org/en/latest/topics/item-pipeline.html

import json

from itemadapter import ItemAdapter

spider_count = 2

class JsonWriterPipeline:
    file = open('items.json', 'w')

    def open_spider(self, spider):
        return None

    def close_spider(self, spider):
        global spider_count
        spider_count -= 1
        if spider_count == 0:
            self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(ItemAdapter(item).asdict()) + "\n"
        self.file.write(line)
        return item