如何通过每一行处理大的json文件并有效地转换为csv？

Question

I have a big file with json string in each row and I want to select few attributes and save those as csv. I have the following code for that. There is around 2 million rows and I want to extract part of them, up to 1 million. I know that better solution would be to store jsons not in one file, so maybe there is a way to split it first?

Anyway, having code like this reaches memory limit when trying to make a dataframe at once. Could you suggest the most appropriate solution, please?

import json
import pandas as pd
from itertools import islice
from pandas.io.json import json_normalize

data = []

with open("input.json") as f:
    head = list(islice(f, 0, 100000))

    for line in head:
        if 'cat' in line:
            data.append(json.loads(line))

#creating a dataframe 
df = json_normalize(data, errors='ignore')
df = df[['col1','col2','col3']]
df.to_csv("output.csv", header=True, sep=';')

Answer 1

why you are keeping head in the memory. You can directly use the loop and stop when you reach 1 million.