如何使用 Python 将 Web 服务的非常大的 JSON 响应转换为 CSV？

Question

I call a web service that returns a very large JSON response. I want to parse it and convert it into a CSV format using Python. I have written a code to load json and convert it to CSV. However, for a large response it raises MemoryError. How can I load and convert response data using streaming?

Here is my code

import json
from pandas import json_normalize
re = requests.get(url)
data = json.loads(re.text)
df = json_normalize(data)
df.to_csv(fileName, index=False, encoding='utf-8')

Here is a sample of my JSON response:

[{"F1":"V1_1","F2":false,,"F3":120,"F4":"URL1","F5":{"F5_1":4,"F5_2":"A"}},
{"F1":"V2_1","F2":true,,"F3":450,"F4":"URL2","F5":{"F5_1":13,"F5_2":"B"}},
{"F1":"V3_1","F2":false,,"F3":312,"F4":"URL3","F5":{"F5_1":6,"F5_2":"C"}},
...
]

The MemoryError occurs in the json.loads() function. I also test following python code:

import pandas as pd
response = requests.get(url)
data = response.json()
df = pd.json_normalize(data)
df.to_csv("filename.csv", index=False, encoding="utf-8")

But still there is a MemmoryError on response.json() function. Is there any idea how I can load and parse and convert such a big JSON response to a CSV file?

Answer 1

There is no well-known or "the best" way to handle very large JSON files.

However requests library provide a way to stream results line by line and with modifications of lines it might be possible to achieve your task.

The algorithm is simple:

• Iterate over these lines parsing every single line as JSON as in streaming results example in requests library
• replace "special" markers in the data stream so JSON parser could parse each record without problems. Markers: delete openning [ , replace separator },\\n{ by }\\n{ , delete closing ] . For your example you'll need to replace double coma ,, with single one , as well

At the end you should receive code resembling the following:

import requests

import json, pandas as pd

url = '...'
filename = '...'

def decode_record(r):
    pl: str = r.decode('utf-8')
    pl = pl.replace('[{', '{').replace('}},', '}}').replace(',,', ',').replace('}}]', '}}')
    # The rest of cleanup goes here
    return json.loads(pl)


def run():
    r = requests.get(url, stream=True)
    res = []
    for line in r.iter_lines():

        # filter out keep-alive new lines
        if line:
            jso = decode_record(line)
            # You might also want to stream lines directly to CSV file here,
            # just not to allocate the DataFrame
            res.append(jso)
    df = pd.DataFrame(res)
    # Parsingof F5 field may be better performed with Pandas functions
    # because it's still a complex object
    print(df.info())
    df.to_csv(filename, index=False, encoding='utf-8')

The variation without a dataframe:

import requests
import json

url = '...'
filename = '...'

def decode_record(r):
    pl: str = r.decode('utf-8')
    pl = pl.replace('[{', '{').replace('}},', '}}').replace(',,', ',').replace('}}]', '}}')
    # The rest of cleanup goes here
    return json.loads(pl)


def encode_csv_record(jso):
    res = []
    for k,v in jso.items():
        res.append(str(v))
    return ','.join(res)


def run():
    r = requests.get(url, stream=True)
    res = []
    with open(filename, 'w') as csvout:
        for line in r.iter_lines():
            # filter out keep-alive new lines
            if line:
                jso = decode_record(line)
                csv_line = encode_csv_record(jso)
                csvout.writelines(csv_line)

Of cause this answer has some leaks but it should present the idea.

Answer 2

This may also get a memory error, but is a simplified approach.

# Get JSON Data
re = requests.get(url)

# Write to .CSV
f = open(fileName, "w")
with f:
    f.write(re.text)
    f.close()

Using the yield command instead of return statement would allow you to return units of work.