如何从 WEIRD JSON 响应中获取价值

Question

I was trying to get data from this API link: https://www.longandfoster.com/include/ajax/api.aspx?op=SearchAgents&firstname=&lastname=&page=1&pagesize=200

If you go to the above link you will see a weird JSON response. The keys and values are not properly displayed.

I converted the duct response into a list and iterate over it. I got the response but the value against the key is not printing, instead, it's returning None

{'Name': None}

import scrapy import json

class MainSpider(scrapy.Spider):
    name = 'main'
    # allowed_domains = ['longandfoster.com']
    start_urls = ['https://www.longandfoster.com/include/ajax/api.aspx?op=SearchAgents&firstname=&lastname=&page=1&pagesize=200']

    def parse(self, response):
        # resp = json.loads(response.body)
        resp_list = []
        resp = json.loads(response.body)
        resp_list.append(resp)

        for each in resp_list:
            name = each.get('DisplayName')

            yield {
                "Name": name,
            }

Answer 1

You have to use json.loads() two times

 resp = json.loads( json.loads(response.body)['Entity'] )

and then your code works.

---

Minimal working code which you can put in one file and run python script.py without creating project.

import scrapy
import json


class MainSpider(scrapy.Spider):
    
    name = 'main'
    # allowed_domains = ['longandfoster.com']
    start_urls = ['https://www.longandfoster.com/include/ajax/api.aspx?op=SearchAgents&firstname=&lastname=&page=1&pagesize=200']

    def parse(self, response):
        resp = json.loads(json.loads(response.body)['Entity'])
        for each in resp:
            name = each.get('DisplayName')

            yield {
                "Name": name,
            }

# --- run without project and save in `output.csv` ---

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',
    # save in file CSV, JSON or XML
    'FEED_FORMAT': 'csv',     # csv, json, xml
    'FEED_URI': 'output.csv', #
})
c.crawl(MainSpider)
c.start()