How can I import multiple JSON files and dump them on the same CSV using Python?

Question

URLs = ['https://cornershopapp.com/api/v2/stores?locality=+01020&country=MX', 
'https://cornershopapp.com/api/v2/stores?locality=+01110&country=MX', 
'https://cornershopapp.com/api/v2/stores?locality=+01210&country=MX']

I have this list of endpoints (above it's just an example - the real list is much bigger). I would like to get the data of these JSON files and append them to one CSV .

Answer 1

here an example with python3

import csv, json, sys
import urllib.request

outputFile = open("test.csv", 'w') #load csv file
with urllib.request.urlopen("https://cornershopapp.com/api/v2/stores?locality=+01020&country=MX") as url:
    data = json.loads(url.read().decode())
    #print (data)
output = csv.writer(outputFile) #create a csv.write
output.writerow(data[0].keys())  # header row
for row in data:
    output.writerow(row.values()) #values row

Answer 2

The data structure for those sample links is hierarchical, so it will be non-trivial to flatten that to a CSV structure. There are lots of ways to do it, but I don't think this is the type of question you can just answer quickly on stackoverflow. Maybe somebody else knows about a library I don't, but I would recommend reading " How to Flatten Deeply Nested JSON Objects in Non-Recursive Elegant Python ".

Answer 3

This snippet utilizes the requests library, which is the de-facto Python library for handling anything HTTP-related.

This script will dump the JSON of the URLs in URLs to output.csv in the current working directory.

EDIT: as others have pointed out, if you'd like to unnest the JSON of these endpoints you'll need to do a bit more work and I'd recommend checking out pandas and the json_normalize function OR study up on recursion and implement your own un-nesting function

import json
import csv

import requests


URLs = ['https://cornershopapp.com/api/v2/stores?locality=+01020&country=MX',
        'https://cornershopapp.com/api/v2/stores?locality=+01110&country=MX',
        'https://cornershopapp.com/api/v2/stores?locality=+01210&country=MX']


with open('output.csv', 'w') as url_file:
    # create CSV writer object
    url_writer = csv.writer(url_file, delimiter=',')

    for u in URLs:
        response = requests.get(u).text
        json_response = json.loads(response)
        url_writer.writerow(json_response)