how to write multiple JSON files into one csv?

Question

I can't find any answers(

i have parsed json from web, this syntax {"a" : 1, "b" : 2, "c" : 3} and transform it to dictonary.

url = 'some.url/1'
data = requests.get(url=url)

binary = data.content

output = json.loads(binary)

There i have regular dictonary. And now i need to write "a" and "c" into csv file, but the most difficult problem is that I must have many iterations of this function, ie from some.url/1 to some.url/100, and all output data must be in one file, how do I do this?

Thank you so much

Answer 1

Using DictWriter , assuming that all the document have a consistent structure:

import csv
import requests

with open('outfile.csv', 'wb') as outfile:
    fieldnames = ['a', 'b', 'c']
    writer = csv.DictWriter(outfile, fieldnames=fieldnames)

    # write the headers, optionally
    writer.writeheader()

    for i in range(1, 101):
        # do you stuff to fetch data
        url = 'some.url/%i' % i
        data = requests.get(url=url)
        binary = data.content
        output = json.loads(binary)

        # now write
        writer.writerow(output)

Answer 2

I'm just going to post an example where you keep json:

import random
import json

output = {}

for url in ["url1","url2"]:
    r = {"a" : random.randint(1,10), "b" : random.randint(1,10), "c" : random.randint(1,10)}
    del r["c"]
    output[url] = r

json.dumps(output)


#with open("output.json","w") as f:
#    f.write(json.dumps(output))

Returns

'{"url1": {"a": 10, "b": 4}, "url2": {"a": 6, "b": 7}}'

---

This can be loaded into a pandas dataframe:

import pandas as pd
df = pd.read_json("output.json").T
print(df)

        a   b
url1    10  6
url2    10  5

From here you can do all kind of operations...

Answer 3

Use a loop that runs from 1 to 100. Append the loop variable to the URL, get the JSON, parse it, and write what you want to the CSV.

with open('output.csv', 'w') as f:
    csvfile = csv.writer(f)
    for i in range(1, 101):
        url = 'some.url/' + str(i)
        data = requests.get(url=url)
        output = json.loads(data.content)
        csvfile.writerow([output['a'], output['c']])