Scrapy output JSON 或 CSV

Question

我正在嘗試使用此代碼settings.py抓取 web

FEED_EXPORT_ENCODING = 'utf-8'

import datetime
now = datetime.datetime.now ()
formatted = now.strftime ("%Y%m%d_%H%M")
FEED_URI = f'\\C:\\Users\\Acer\\Desktop\\{formatted}.csv'
FEED_TYPE = 'csv'

有了這個 special_offers.py

# -*- coding: utf-8 -*-
import scrapy
import datetime


class SpecialOffersSpider(scrapy.Spider):
    name = 'special_offers'
    allowed_domains = ['www.tinydeal.com']

    def start_requests(self):
        yield scrapy.Request(url='https://www.tinydeal.com/specials.html', callback=self.parse, headers={
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
        })

    def parse(self, response):
        for product in response.xpath("//ul[@class='productlisting-ul']/div/li"):
            yield {
                'title': product.xpath(".//a[@class='p_box_title']/text()").get(),
                'url': response.urljoin(product.xpath(".//a[@class='p_box_title']/@href").get()),
                'discounted_price': product.xpath(".//div[@class='p_box_price']/span[1]/text()").get(),
                'original_price': product.xpath(".//div[@class='p_box_price']/span[2]/text()").get(),
                'User-Agent': response.request.headers['User-Agent'].decode('utf-8'),
                'datetime': datetime.datetime.now().strftime("%Y%m%d %H%M")

            }

        next_page = response.xpath("//a[@class='nextPage']/@href").get()

        if next_page:
            yield scrapy.Request(url=next_page, callback=self.parse, headers={
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
            })

然后我打開終端並使用

scrapy crawl special_offers

問題是，當我導出 JSON 時，數據在 }{ 之間沒有逗號。 例如，使 Power BI 無法讀取我的文件

當我導出 CSV 時，數據會有所不同，然后我期望使用 EXCEL 打開時

CSV data example {"title": "ABS Plastic Case for Raspberry Pi 3 Model B & Raspberry Pi 2 E-524988", "url": " https://www.tinydeal.com/abs-plastic-case-for- raspberry-pi-3-model-b-raspberry-pi-2-p-163950.html ", "discounted_price": "R$12.74", "original_price": "R$13.66", "User-Agent": "Mozilla/ 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"} {"title": "3M 9001 KN90 Dust Masks Respirator防塵 PM2.5 工業建築 Polle RTH-562440", "url": " https://www.tinydeal.com/3m-9001-kn90-dust-masks-respirator-anti-dust-pm25-industrial-construction -polle-p-179487.html ”，“discounted_price”：“R$10.29”，“original_price”：“R$12.40”，“用戶代理”：“Mozilla/5.0（Windows NT 10.0；Win64；x64）AppleWebKit/537.36 （KHTML，如 Gec ko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"} {"title": "2-in-1 Vintage 藍色水鑽項鏈 + 耳環首飾套裝 DJA-562974", "url": " https://www.tinydeal.com/2-in-1-vintage-blue-rhinestone-necklace-earring-jewelry-set-p-180097.html ", "discounted_price": "R$1. “R$30.77”，“用戶代理”：“Mozilla/5.0（Windows NT 10.0； 贏64； x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"} {"title": "64GB USB 2.0 Flash Drive USB Pen Drive U Disk EFM-561923" , "url": " https://www.tinydeal.com/64gb-usb-20-flash-drive-usb-pen-drive-u-disk-p-178875.ZFC35FDC70D5FC69D2698$83A822C7A53EZ ", "discounted38_price:". ", "original_price": "R$99.43", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "日期時間”：“20200420 2330”}

JSON 數據示例

{ "title": "用於 Raspberry Pi 3 Model B & Raspberry Pi 2 E-524988 的 ABS 塑料外殼", "url": " https://www.tinydeal.com/abs-plastic-case-for-raspberry-pi -3-model-b-raspberry-pi-2-p-163950.html ", "discounted_price": "R$12.74", "original_price": "R$13.66", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2329" } { "title": "3M 9001 KN90 Dust Masks Respirator Anti-dust PM2.5 工業建築Polle RTH-562440", "url":" https://www.tinydeal.com/3m-9001-kn90-dust-masks-respirator-anti-dust-pm25-industrial-construction-polle- p-179487.html ", "discounted_price": "R$10.29", "original_price": "R$12.40", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,像壁虎）Chrome/76.0.3809.100 Safari/537.36", "dat etime": "20200420 2329" } { "title": "二合一復古藍色水鑽項鏈 + 耳環首飾套裝 DJA-562974", "url": " https://www.tinydeal.com/2-in -1-vintage-blue-rhinestone-necklace-earring-jewelry-set-p-180097.html ”，“discounted_price”：“R$11.77”，“original_price”：“R$30.77”，“User-Agent”：“Mozilla /5.0（Windows NT 10.0； 贏64； x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2329" }

誰能告訴我這些輸出哪里出錯了？

Answer 1

您如何獲取抓取的數據？ 根據您顯示的內容，我懷疑您是從終端復制的。 是這樣嗎？ 如果是，則可以使用以下命令將其直接保存到文件中：

scrapy crawl special_offers -o <where save the file>/special_offers.json

希望這可以解決您的問題。 請告訴我。