如何在Scrapy中使用包含URL和ID的JSON并构造结果？

Question

I'm using Scrapy to scrape datas from 9000+ urls contained in a JSON with a matching ID. Here is my JSON object type:

[{
"objectID": 10500,
"gm_url": "https://reddit.com/1"
},
"objectID": 10501,
"gm_url": "https://reddit.com/2"
}]

I'd like to have my results in a json with the scraped datas, the matching url and the id.

[{
    "objectID": 10500,
    "gm_url": "https://reddit.com",
    "results": [
        {
            "model": "",
            "price": "",
            "auction": "",
            "date": "",
            "auction_url": "",
            "img": ""
        },
        {
            "model": "",
            "price": "",
            "auction": "",
            "date": "",
            "auction_url": "",
            "img": ""
        },
        {
            "model": "",
            "price": "",
            "auction": "",
            "date": "",
            "auction_url": "",
            "img": ""
        }
    ]
}]

Here is my code right now in scrapy (which is kind of messy):

import json
import scrapy

with open('/home/bolgi/Workspace/Dev/python_workspace/gm_spider/Json/db_urls_glenmarch_results_scrapy_reduced.json', encoding='utf-8') as data_file:
    data = json.load(data_file)

for item in data:
    objectId = item['objectID']
    gmUrl = item['gm_url']

    class GlenMarchSpider(scrapy.Spider):
        name = 'glenmarch'

        def start_requests(self):
            start_urls = gmUrl

            for url in start_urls:
                yield scrapy.Request(url, callback=self.parse)

        def parse(self, response):
            for caritem in response.css("div.car-item-border"):
                yield {
                    "url": response.url,
                    "model": caritem.css("div.make::text").get(),
                    "price": caritem.css("div.price::text").get(),
                    "auction": caritem.css("div.auctionHouse::text").get(),
                    "date": caritem.css("div.date::text").get(),
                    "auction_url": caritem.css("div.view-auction a::attr(href)").get(),
                    "img": caritem.css("img.img-responsive::attr(src)").get()
                }

I don't know how to structure the code and how to use the JSON file, I'm new in python and it's a bit difficult for me.

Answer 1

You should never declare a class inside a forloop.

I suggest you the following structure:

import json
import scrapy

class GlenMarchSpider(scrapy.Spider):
    name = 'glenmarch'

    def __init__(self):
        with open('/home/bolgi/Workspace/Dev/python_workspace/gm_spider/Json/db_urls_glenmarch_results_scrapy_reduced.json', encoding='utf-8') as data_file:
            self.data = json.load(data_file)

    def start_requests(self):
        for item in self.data:
            request = scrapy.Request(item['gm_url'], callback=self.parse)
            request.meta['item'] = item
            yield request

    def parse(self, response):
        item = response.meta['item']
        item['results'] = []
        for caritem in response.css("div.car-item-border"):
            item['results'].append({
                "model": caritem.css("div.make::text").get(),
                "price": caritem.css("div.price::text").get(),
                "auction": caritem.css("div.auctionHouse::text").get(),
                "date": caritem.css("div.date::text").get(),
                "auction_url": caritem.css("div.view-auction a::attr(href)").get(),
                "img": caritem.css("img.img-responsive::attr(src)").get()
            })
        yield item

Then you can call your spider (and save it in a new json file):

$ scrapy crawl glenmarch -o myjson.json -t json

If there are things in the code that you do not understand do not hesitate to ask for clarifications ! :)

Answer 2

The scrapy.spider also has start_urls list field which in default empty where you can append all the urls .

import scrapy
import json

class GlenMarchSpider(scrapy.Spider)
    name = 'glenmarch'
    start_urls = []

    with open('/home/bolgi/Workspace/Dev/python_workspace/gm_spider/Json/db_urls_glenmarch_results_scrapy_reduced.json', encoding='utf-8') as json_file:
         data = json.load(json_file)
         for item in data:
             objectId = item['objectID']
             gmUrl = item['gm_url']
             start_urls.append(gmUrl)

    def parse(self, response):
        item = {}
        for caritem in response.css("div.car-item-border"):
            yield {
                "url": response.url,
                "model": caritem.css("div.make::text").get(),
                "price": caritem.css("div.price::text").get(),
                "auction": caritem.css("div.auctionHouse::text").get(),
                "date": caritem.css("div.date::text").get(),
                "auction_url": caritem.css("div.view-auction a::attr(href)").get(),
                "img": caritem.css("img.img-responsive::attr(src)").get()
            }

And you can run the spider in that way too:

scrapy runspider quotes_spider.py -o glenmarch.json

For more details, please check out the official document or feel free to ask. https://scrapy.readthedocs.io/en/latest/intro/overview.html